Garbage collection working improperly?

[Oliver Walczak]

Dear List,
Trying the following hack:
[color=blue][color=green][color=darkred]
>>> a = []
>>> for i in range(0,9999999 ):[/color][/color][/color]
a.append(i)[color=blue][color=green][color=darkred]
>>> del(a)[/color][/color][/color]

Builds up a great list in memory and immediately deletes it. Unfortunately
the task manager shows me that i allocated about 155MB in memory before
del(), but del only releases about 40MB of them so i'm leaving about 117 MB
of reserved memory after deleting the list.
I'm using python 2.2.3 on WinXP.
Any ideas about that? How can i dealloc the left memory space?
Best regards
Oliver

[Diez B. Roggisch]

Re: Garbage collection working improperly?

Hi,
[color=blue]
> Builds up a great list in memory and immediately deletes it. Unfortunately[/color]

if all you want is to create a loop, use xrange instead of range - it
won't create that huge list.
[color=blue]
> the task manager shows me that i allocated about 155MB in memory before
> del(), but del only releases about 40MB of them so i'm leaving about 117 MB
> of reserved memory after deleting the list.
> I'm using python 2.2.3 on WinXP.
> Any ideas about that? How can i dealloc the left memory space?[/color]

Nope, no idea - the only thing that I can think of is that python not
necessarily releaseses the memory because it uses its own allocation
scheme. I don't know that for sure, but I definitely would go for such a
thingy if I was to implement a virtual machine.
Its like in JAVA - each object instantiation and destruction isn't
paired with system malloc/free calls, but some internal memory manager
deals with that. And if it rans out of space, it will request more from
the system.

Regards,

Diez

[Duncan Booth]

Re: Garbage collection working improperly?

"Oliver Walczak" <oliver.walczak @momatec.de> wrote in
news:mailman.95 3.1069412317.70 2.python-list@python.org :
[color=blue]
> Builds up a great list in memory and immediately deletes it.
> Unfortunately the task manager shows me that i allocated about 155MB
> in memory before del(), but del only releases about 40MB of them so
> i'm leaving about 117 MB of reserved memory after deleting the list.
> I'm using python 2.2.3 on WinXP.
> Any ideas about that? How can i dealloc the left memory space?[/color]

You cannot, but it shouldn't matter. If you aren't using it, then it will
eventually get paged out and occupy space in the swapfile, but the physical
memory will be allocated to other processed.

Small objects get handled specially by Python's memory allocation, and the
memory used for small objects will never be returned to the operating
system (until the process terminates). Its a trade off, because it means
that allocating the small objects is much faster, but the cost is extra
swap space used, not extra physical memory.

If it really matters to you to allocate a massive spike of memory then
release it all, you could try extracting the memory hungry code out into a
separate process that can terminate once it has finished whatever
calculation you are doing. That way all the memory really would be
released.

--
Duncan Booth duncan@rcp.co.u k
int month(char *p){return(1248 64/((p[0]+p[1]-p[2]&0x1f)+1)%12 )["\5\x8\3"
"\6\7\xb\1\x9\x a\2\0\4"];} // Who said my code was obscure?

[Jay O'Connor]

Re: Garbage collection working improperly?

Diez B. Roggisch wrote:
[color=blue]
> Hi,
>[color=green]
>> Builds up a great list in memory and immediately deletes it.
>> Unfortunately[/color]
>
>
> if all you want is to create a loop, use xrange instead of range - it
> won't create that huge list.
>[color=green]
>> the task manager shows me that i allocated about 155MB in memory before
>> del(), but del only releases about 40MB of them so i'm leaving about
>> 117 MB
>> of reserved memory after deleting the list.
>> I'm using python 2.2.3 on WinXP.
>> Any ideas about that? How can i dealloc the left memory space?[/color]
>
>
> Nope, no idea - the only thing that I can think of is that python not
> necessarily releaseses the memory because it uses its own allocation
> scheme. I don't know that for sure, but I definitely would go for such
> a thingy if I was to implement a virtual machine.
> Its like in JAVA - each object instantiation and destruction isn't
> paired with system malloc/free calls, but some internal memory manager
> deals with that. And if it rans out of space, it will request more
> from the system.[/color]



I kinda like VisualWorks Smalltalk's approach. It uses a system whereby
new objects are scavenged aggresivly and objects that survive the
scavenger are moved to a different memory space that is not GC'ed nearly
as much. (New objects tend to have short lives..objects that survive
several generations of scavenging tend to live longer) When it's memory
reaches a threshold, it makes a decision as to whether to either request
more from the OS or to GC the old object memory space in an attempt to
free more memory. The nice part is that you can configure a. how much
memory it starts with b. at what point the threshold is set for and
c.the weighting of the algorithim to determine whether to GC or allocate


Take care,
Jay

[Dennis Lee Bieber]

Re: Garbage collection working improperly?

Oliver Walczak fed this fish to the penguins on Friday 21 November 2003
02:58 am:

[color=blue]
>
> Builds up a great list in memory and immediately deletes it.
> Unfortunately the task manager shows me that i allocated about 155MB
> in memory before del(), but del only releases about 40MB of them so
> i'm leaving about 117 MB of reserved memory after deleting the list.
> I'm using python 2.2.3 on WinXP.
> Any ideas about that? How can i dealloc the left memory space?[/color]

I suspect the initial allocation is the OS assigning memory to the
process space. The del() is flagging that memory as free for reuse, but
not many OS runtimes are equipped to return freed memory to the OS (ie,
remove it from the process space). The stray 40MB may have been OS
related pointers to the occupied segments (if they had been allocated
in small chunks rather than one solid block); after the del() those
segment tracking pointers may have been freed to the OS, leaving just a
process header saying the process address space is n-MB long.

--[color=blue]
> =============== =============== =============== =============== == <
> wlfraed@ix.netc om.com | Wulfraed Dennis Lee Bieber KD6MOG <
> wulfraed@dm.net | Bestiaria Support Staff <
> =============== =============== =============== =============== == <
> Bestiaria Home Page: http://www.beastie.dm.net/ <
> Home Page: http://www.dm.net/~wulfraed/ <[/color]

[Aahz]

Re: Garbage collection working improperly?

In article <bplcb9$f4g$1@r eader2.nmix.net >,
Jay O'Connor <joconnor@cyber mesa.com> wrote:[color=blue]
>
>I kinda like VisualWorks Smalltalk's approach. It uses a system whereby
>new objects are scavenged aggresivly and objects that survive the
>scavenger are moved to a different memory space that is not GC'ed nearly
>as much. (New objects tend to have short lives..objects that survive
>several generations of scavenging tend to live longer) When it's memory
>reaches a threshold, it makes a decision as to whether to either request
>more from the OS or to GC the old object memory space in an attempt to
>free more memory. The nice part is that you can configure a. how much
>memory it starts with b. at what point the threshold is set for and
>c.the weighting of the algorithim to determine whether to GC or allocate[/color]

Python does the same thing -- you just can't configure it as much.
--
Aahz (aahz@pythoncra ft.com) <*> http://www.pythoncraft.com/

Weinberg's Second Law: If builders built buildings the way programmers wrote
programs, then the first woodpecker that came along would destroy civilization.

[Jay O'Connor]

Re: Garbage collection working improperly?

Aahz wrote:
[color=blue]
>In article <bplcb9$f4g$1@r eader2.nmix.net >,
>Jay O'Connor <joconnor@cyber mesa.com> wrote:
>
>[color=green]
>>I kinda like VisualWorks Smalltalk's approach. It uses a system whereby
>>new objects are scavenged aggresivly and objects that survive the
>>scavenger are moved to a different memory space that is not GC'ed nearly
>>as much. (New objects tend to have short lives..objects that survive
>>several generations of scavenging tend to live longer) When it's memory
>>reaches a threshold, it makes a decision as to whether to either request
>>more from the OS or to GC the old object memory space in an attempt to
>>free more memory. The nice part is that you can configure a. how much
>>memory it starts with b. at what point the threshold is set for and
>>c.the weighting of the algorithim to determine whether to GC or allocate
>>
>>[/color]
>
>Python does the same thing -- you just can't configure it as much.
>
>[/color]

I remember this in particular because I was working on an application
where the performance characteristics were such that for the most part
we wanted to allocate new memory rather than GC, but then as the memory
we were using reached a certain point, we wanted to free some of our own
cached objects so tha the GC could reclaim them. Being able to fine
tune the memory usage characteristics , and also hooking into
VisualWork's notification mechanism to be informed when GC was
happening, proved very important.

[Aahz]

Re: Garbage collection working improperly?

In article <bpli8b$uaf$1@r eader2.nmix.net >,
Jay O'Connor <joconnor@cyber mesa.com> wrote:[color=blue]
>Aahz wrote:[color=green]
>>In article <bplcb9$f4g$1@r eader2.nmix.net >,
>>Jay O'Connor <joconnor@cyber mesa.com> wrote:[color=darkred]
>>>
>>>I kinda like VisualWorks Smalltalk's approach. It uses a system whereby
>>>new objects are scavenged aggresivly and objects that survive the
>>>scavenger are moved to a different memory space that is not GC'ed nearly
>>>as much. (New objects tend to have short lives..objects that survive
>>>several generations of scavenging tend to live longer) When it's memory
>>>reaches a threshold, it makes a decision as to whether to either request
>>>more from the OS or to GC the old object memory space in an attempt to
>>>free more memory. The nice part is that you can configure a. how much
>>>memory it starts with b. at what point the threshold is set for and
>>>c.the weighting of the algorithim to determine whether to GC or allocate[/color]
>>
>>Python does the same thing -- you just can't configure it as much.[/color]
>
>I remember this in particular because I was working on an application
>where the performance characteristics were such that for the most part
>we wanted to allocate new memory rather than GC, but then as the memory
>we were using reached a certain point, we wanted to free some of our own
>cached objects so tha the GC could reclaim them. Being able to fine
>tune the memory usage characteristics , and also hooking into
>VisualWork's notification mechanism to be informed when GC was
>happening, proved very important.[/color]

Don't forget Python uses a very different memory management system. For
starters, Python's primary mechanism is refcounting rather than GC.
More than that, Python now uses PyMalloc as its standard allocation
system, which does a lot to reduce memory fragmentation. You can adjust
the frequency of GC in Python, but that's about it -- and you don't
really need much more than that, I think.
--
Aahz (aahz@pythoncra ft.com) <*> http://www.pythoncraft.com/

Weinberg's Second Law: If builders built buildings the way programmers wrote
programs, then the first woodpecker that came along would destroy civilization.