Python memory handling

[Andrew MacIntyre]

Re: Python memory handling

frederic.pica@g mail.com wrote:

Using the same file with cElementTree took me 217 Mb, with no
unreachable object.
For me it's not a good behavior, it's not a good way to let the system
swap this unused memory instead of freeing it.
I think it's a really good idea to have a memory pool for performance
reason, but why is there no 'free block' limit ?
Python is a really really good language that can do many things in a
clear, easier and performance way I think. It has always feet all my
needs. But I can't imagine there is no good solution for that problem,
by limiting the free block pool size or best, letting the user specify
this limit and even better, letting the user completely freeing it
(with also the limit manual specification)
>
Like:
import pool
pool.free()
pool.limit(size in megabytes)
>
Why not letting the user choosing that, why not giving the user more
flexibility ?

Because its not easy, and its an unusual edge case that hasn't attracted
developer effort (the PyMalloc change for 2.5 was contributed by someone
who desperately needed it, not a core Python developer; it was also a
non-trivial effort to get right).

You should also appreciate something about PyMalloc: it only handles
allocation requests of 256 bytes or smaller, and this limitation is part
of PyMalloc's design.

If most of your allocations are >256 bytes, you're at the mercy of the
platform malloc and heap fragmentation can be a killer. This is probably
why the getlines() approach mentioned would appear to relinquish (most
of) the memory: the list was probably comprised mostly of PyMalloc
allocations.

I haven't checked, but cElementTree may internally not be using PyMalloc
anyway, as the package is stated to be usable back to Python 1.5 - long
before the current allocation management came into effect. In which
case, you're at the mercy of the platform malloc... The pure Python
ElementTree might play more your way, at a performance cost.

--
-------------------------------------------------------------------------
Andrew I MacIntyre "These thoughts are mine alone..."
E-mail: andymac@bullsey e.apana.org.au (pref) | Snail: PO Box 370
andymac@pcug.or g.au (alt) | Belconnen ACT 2616
Web: http://www.andymac.org/ | Australia

[Nick Craig-Wood]

Re: Python memory handling

Andrew MacIntyre <andymac@bullse ye.apana.org.au wrote:

You should also appreciate something about PyMalloc: it only handles
allocation requests of 256 bytes or smaller, and this limitation is part
of PyMalloc's design.
>
If most of your allocations are >256 bytes, you're at the mercy of the
platform malloc

You can tweak this if you are using libc (eg under linux) at least :-

http://www.gnu.org/software/libc/manual/html_node/Malloc-Tunable-Parameters.html

Setting M_MMAP_THRESHOL D should result in blocks that are perfectly
free()able back to the OS if allocated with malloc(). By default this
is 128k I think so you can set it to 4k and it should help a lot.

Note that a mmap block is a minimum of 4k (under x86 - one OS page
anyway) so set this too small and you program will use a *lot* of
memory, but only temporarily ;-)

If PyMalloc stretched up to 4k and M_MMAP_THRESHOL D was set to 4k then
you'd have the perfect memory allocator...

--
Nick Craig-Wood <nick@craig-wood.com-- http://www.craig-wood.com/nick

[=?iso-8859-1?B?RnLpZOlyaWMgUElDQQ==?=]

Re: Python memory handling

Greets,

Sorry for my late answer, google groups lost my post...
First, thanks you for your explanations about memory handling in the
os and python.
I've tried with python 2.5 under linux :
For the parsing of a 66 Mb xml file with cElementTree :
When starting python : 2.1 Mb private memory used
import xml.etree.cElem entTree as ElementTree #3.4 Mb used
et=ElementTree. parse('otherdat a.xml') #218.6 Mb used
del et #43.3 Mb used
et=ElementTree. parse('otherdat a.xml') #218.6 Mb used
del et #60.6 Mb used
et=ElementTree. parse('otherdat a.xml') #218.6 Mb used
del et #54.1 Mb used
et=ElementTree. parse('otherdat a.xml') #218.6 Mb used
del et #54.1 Mb used
et=ElementTree. parse('otherdat a.xml') #218.6 Mb used
del et #54.1 Mb used

Why does I have a such erratic memory freeing ?
I've tried the same test many time with a new interpreter and I've got
43.3 Mb after the first free and 54.1 Mb after the others.
If there is a memory pool limit in list ans dict, why can't I goes
back to 43.3 or 54.1 Mb all the times ?

I've tried using readlines():
When starting python : 2.1 Mb private memory used
f=open('otherda ta.xml') #2.2 Mb used
data=f.readline s() #113 Mb used
del data #2.7 Mb used
f.seek(0) #2.7 Mb used
data=f.readline s() #113 Mb used
del data #2.7 Mb used

That time I have a good memory handling (for my definition of memory
handling)

So is there a problem with cElementTree ?

I've done a last test with ElementTree :
When starting python : 2.1 Mb private memory used
import xml.etree.Eleme ntTree as ElementTree #3.2 Mb used
et=ElementTree. parse('otherdat a.xml') #211.4 Mb used (but very
slow :p)
del et #21.4 Mb used
et=ElementTree. parse('otherdat a.xml') #211.4 Mb used
del et #29.8 Mb used

So why does I have such differences in memory freeing ? Only due to
fragmentation ?

Anyway, python 2.5 has a better memory handling than 2.4, but still
not perfect for me.
I think I've not really understood the problem with the use of malloc
(fragmentation, ...)

Thanks for your help
Regards,
FP