32-bit python memory limits?

[Gary Robinson]

I'm running a Python job on OS X 10.5.3 and the Python 2.5.2 that's available as a binary download at python.org for OS X.

I ran a python program tonight that ended up using much more memory than anticipated. It just kept on using more and more memory. Instead of killing it, I just watched it, using Activity Monitor. I assumed that when it had 2GB allocated it would blow up, because I thought 32-bit python could only address 2GB.

But Activity Monitor reported that it had allocated 3.99GB of virtual memory before it finally blew up with malloc errors. Was my understanding of a 2GB limit wrong? I guess so! But I'm pretty sure I saw it max out at 2GB on linux...

Anybody have an explanation, or is it just that my understanding of a 2GB limit was wrong? Or was it perhaps right for earlier versions, or on linux...??

Thanks for any thoughts,
Gary


--

Gary Robinson
CTO
Emergent Music, LLC
personal email: garyrob@mac.com
work email: grobinson@emerg entmusic.com
Company: http://www.emergentmusic.com
Blog: http://www.garyrobinson.net

[Larry Bates]

Re: 32-bit python memory limits?

Gary Robinson wrote:

I'm running a Python job on OS X 10.5.3 and the Python 2.5.2 that's available as a binary download at python.org for OS X.
>
I ran a python program tonight that ended up using much more memory than anticipated. It just kept on using more and more memory. Instead of killing it, I just watched it, using Activity Monitor. I assumed that when it had 2GB allocated it would blow up, because I thought 32-bit python could only address 2GB.
>
But Activity Monitor reported that it had allocated 3.99GB of virtual memory before it finally blew up with malloc errors. Was my understanding of a 2GB limit wrong? I guess so! But I'm pretty sure I saw it max out at 2GB on linux...
>
Anybody have an explanation, or is it just that my understanding of a 2GB limit was wrong? Or was it perhaps right for earlier versions, or on linux...??
>
Thanks for any thoughts,
Gary
>
>

Yep, 2Gb is only 31 bits. 4Gb is 32 bits (since memory address is an unsigned).

-Larry

[Grant Edwards]

Re: 32-bit python memory limits?

On 2008-06-24, Gary Robinson <garyrob@mac.co mwrote:

I'm running a Python job on OS X 10.5.3 and the Python 2.5.2
that's available as a binary download at python.org for OS X.
>
I ran a python program tonight that ended up using much more
memory than anticipated. It just kept on using more and more
memory. Instead of killing it, I just watched it, using
Activity Monitor. I assumed that when it had 2GB allocated it
would blow up, because I thought 32-bit python could only
address 2GB.

You assumed incorrectly. :)

But Activity Monitor reported that it had allocated 3.99GB of
virtual memory before it finally blew up with malloc errors.
Was my understanding of a 2GB limit wrong? I guess so! But I'm
pretty sure I saw it max out at 2GB on linux...

You may have been using a Linux kernel with a 2GB user VM
limit.

Anybody have an explanation, or is it just that my
understanding of a 2GB limit was wrong? Or was it perhaps
right for earlier versions, or on linux...??

In the past, it was pretty common for Linux kernels to be
configured so that a user-space program had 2GB of virtual
address space. For reasons that aren't really important here,
doing so made it simpler to manage various kernel vs. user
memory issues. Back when nobody could afford more than a few
hundred MB of RAM, that made sense. These days, the Linux
kernel can be built for 2GB, 3GB or 4GB of VM space. There's a
tiny bit of extra overhead if you pick 3GB or 4GB, so users
with 2GB or less of physical RAM still often configure kernels
for a 2GB limit.

http://kerneltrap.org/node/2450

--
Grant Edwards grante Yow! Can you MAIL a BEAN
at CAKE?
visi.com