Python memory deallocate

[mardif]

Re: Python memory deallocate

OK! i will test my app with python 2.5a2

thx

[Bruno Desthuilliers]

Re: Python memory deallocate

mardif a écrit :[color=blue]
> OK! i will test my app with python 2.5a2[/color]

Please let us know the result.

[Fredrik Lundh]

Re: Python memory deallocate

bruno at modulix" wrote:
[color=blue]
> *please* re-read carefully what I and Diez wrote earlier in this thread
> before jumping to possibly erroneous conclusion. I didn't say that the
> problem *actually* was with Python - just that it *may* have to do with
> a memory management issue fixed in 2.5.[/color]

the only thing that has changed is that Python 2.5 is slightly more likely to release
memory areas held by the Python object allocator back to the C runtime memory
allocator. if you have a leak in your application, changing to Python 2.5 won't
change a thing.

</F>

[Fredrik Lundh]

Re: Python memory deallocate

Heiko Wundram wrote:
[color=blue]
> PIL isn't known to have any memory leaks, by the way (AFAICT), just to confirm
> what I've written before, but the effbot should be of more help here...[/color]

PIL is heavily used in 24/7 production systems, often by people who knows a lot
about how to run mission critical systems, so memory and resource leaks in PIL
tends to be noticed.

there has been one leak fix in 1.1.6, afaik: converting a grayscale image to a palette
image would (sometimes?) leak a palette structure. but that's a couple of hundred
bytes, not a couple of hundred megabytes...

</F>

[bruno at modulix]

Re: Python memory deallocate

Fredrik Lundh wrote:[color=blue]
> bruno at modulix" wrote:
>
>[color=green]
>>*please* re-read carefully what I and Diez wrote earlier in this thread
>>before jumping to possibly erroneous conclusion. I didn't say that the
>>problem *actually* was with Python - just that it *may* have to do with
>>a memory management issue fixed in 2.5.[/color]
>
>
> the only thing that has changed is that Python 2.5 is slightly more likely to release
> memory areas held by the Python object allocator back to the C runtime memory
> allocator. if you have a leak in your application, changing to Python 2.5 won't
> change a thing.[/color]

Which is mostly what I meant - sorry if it wasn't clear.

--
bruno desthuilliers
python -c "print '@'.join(['.'.join([w[::-1] for w in p.split('.')]) for
p in 'onurb@xiludom. gro'.split('@')])"

[Heiko Wundram]

Re: Python memory deallocate

Am Donnerstag 11 Mai 2006 18:07 schrieb Michele Petrazzo:[color=blue]
> Heiko Wundram wrote:[color=green]
> > As was said before: as long as you keep a reference to an object, the
> > object's storage _will not be_ reused by Python for any other objects
> > (which is sensible, or would you like your object to be overwritten by
> > other objects before you're done with them?). Besides, even if Python did
> > free the memory that was used, the operating system wouldn't pick it up
> > (in the general case) anyway (because of fragmentation issues), so Python
> > keeping the memory in an internal free-list for new objects is a sensible
> > choice the Python developers took here.[/color]
>
> This isn't true. Just tried with python 2.5a2 and:
>
> d:\python25\pyt hon.exe
>[color=green][color=darkred]
> >>> a = range(1000 * 100 *100) # 173 MB
> >>> del a # 122MB[/color][/color]
>
> So now, like you saied, if I try to allocate another memory chunk,
>
> python'll re-use it... But this isn't true:[color=green][color=darkred]
> >>> b = range(100 * 100 * 100) # 126 MB
> >>> del b # 122MB
> >>> exit() # :)[/color][/color]
>
> d:\python25\pyt hon.exe
>[color=green][color=darkred]
> >>> b = range(100 * 100 * 100) # 19 MB[/color][/color]
>
> Do why python don't reuse the freed memory and re-allocate 4 MB (126 -
> 122)?[/color]

What the OS reports as the memory usage isn't actually what's allocated by
Python objects. As Tim Peters pointed out in another post, this number just
specifies the amount of memory the OS has given to libc, which in turn has
given it to Python.

Basically, what you're seeing is exactly the reuse of memory that Python
internally does. Why it allocates 4 more megabytes I wouldn't know, but as
you can see, the range b is actually 19 megabytes in size (a little smaller,
but >10MB), whereas it only takes up 4 megabytes when you construct it after
you deleted the old list. That's a major difference.

I just tried the following:

modelnine@phoen ix ~ $ python
Python 2.4.3 (#1, May 4 2006, 23:51:29)
[GCC 3.4.6 (Gentoo 3.4.6-r1, ssp-3.4.5-1.0, pie-8.7.9)] on linux2
Type "help", "copyright" , "credits" or "license" for more information.[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

1000 7836 0.0 0.3 15816 3540 pts/1 S+ 15:09 0:00 python
[color=blue][color=green][color=darkred]
>>> x = range(1000000)[/color][/color][/color]

1000 7836 0.2 3.8 47656 35388 pts/1 S+ 15:09 0:00 python
[color=blue][color=green][color=darkred]
>>> del x[/color][/color][/color]

1000 7836 0.2 2.9 39840 27572 pts/1 S+ 15:09 0:00 python
[color=blue][color=green][color=darkred]
>>> x = range(1000000)[/color][/color][/color]

1000 7836 0.2 3.8 48720 35460 pts/1 S+ 15:09 0:00 python
[color=blue][color=green][color=darkred]
>>> del x[/color][/color][/color]

1000 7836 0.2 2.9 40904 27644 pts/1 S+ 15:09 0:00 python
[color=blue][color=green][color=darkred]
>>> x = range(1000000)[/color][/color][/color]

1000 7836 0.2 3.8 48720 35460 pts/1 S+ 15:09 0:00 python
[color=blue][color=green][color=darkred]
>>> del x[/color][/color][/color]

1000 7836 0.2 2.9 40904 27644 pts/1 S+ 15:09 0:00 python

....

Ad infinitum (see the pattern?)

Convinced now that Python reuses memory? (see the sixth column for actual
memory usage, the fifth is mapped memory, which contains shared libraries,
etc.) The list is about 32MB in size (see the initial jump from 3MB to 35MB),
but only adds about 8MB in each iteration, which are freed in each run.

--- Heiko.