Python embedded in C & memory releasing

[lebo]

Hi

I'm trying to understand how Python handles memory usage and dynamic
object loading and unloading. Problem to solve? Build a very low
memory footprint (non-GUI) Python application for Unix/Windows. Why
use Python? End user flexibility.

So far it seems that (on Windows):
(1) memory increases when you import <module>. Expected behaviour.
(2) memory does not release when you del <module>. Why not? Is Python
not releasing the memory or is it the OS not releasing memory?

So then I tried embedded Python in C:
(1) c application base 1.6MBytes (a good number)
(2) Call Python function from c application - memory up to 2.6MByte
(OK number)
(3) Python function exits but memory usage stays at 2.6MByte

Why is memory not releasing? How can I tell if this Python behaviour
or OS behaviour?

C code snippet:
#include <Python.h>

int main(int argc, char *argv[])
{
int y = 0;
printf("Started ...\n");
for (int count=0;count < 999999999;count ++) {
y += 1;
}
printf("Initing Py\n");
Py_Initialize() ;
PyRun_SimpleStr ing("from time import sleep\n"
"print 'I will sleep for a while'\n"
"sleep(10)\ n"
"print 'Exiting Py'\n");
Py_Finalize();

for (int count=0;count < 999999999;count ++) {
y += 1;
}
return 0;
}

Thanks, Leonard

[Tom Anderson]

Re: Python embedded in C &amp; memory releasing

On 23 Sep 2003, lebo wrote:
[color=blue]
> I'm trying to understand how Python handles memory usage and dynamic
> object loading and unloading. Problem to solve? Build a very low
> memory footprint (non-GUI) Python application for Unix/Windows. Why
> use Python? End user flexibility.
>
> So far it seems that (on Windows):
> (1) memory increases when you import <module>. Expected behaviour.
> (2) memory does not release when you del <module>. Why not? Is Python
> not releasing the memory or is it the OS not releasing memory?
>
> So then I tried embedded Python in C:
> (1) c application base 1.6MBytes (a good number)
> (2) Call Python function from c application - memory up to 2.6MByte
> (OK number)
> (3) Python function exits but memory usage stays at 2.6MByte
>
> Why is memory not releasing? How can I tell if this Python behaviour
> or OS behaviour?[/color]

this is how most runtime systems handle memory. they have a certain amount
of space allocated, and if they run out, they ask the OS for more, but
they never give it back. i think this is what most implementations of
malloc/free do; it's certainly what java does.

what you want is some way to tell python to give back some or all of its
spare memory. i'm not aware of such a thing myself.

tom

--
unconstrained by any considerations of humanity or decency

[John J. Lee]

Re: Python embedded in C &amp; memory releasing

leonardbocock@y ahoo.com (lebo) writes:
[color=blue]
> I'm trying to understand how Python handles memory usage and dynamic
> object loading and unloading. Problem to solve? Build a very low
> memory footprint (non-GUI) Python application for Unix/Windows. Why
> use Python? End user flexibility.
>
> So far it seems that (on Windows):
> (1) memory increases when you import <module>. Expected behaviour.
> (2) memory does not release when you del <module>. Why not? Is Python
> not releasing the memory or is it the OS not releasing memory?[/color]
[...]

On almost all OSes, processes hold on to all memory that has been
allocated by the OS. Memory released by a process can only be reused
*by that same process*. The OS only gets it back when the process
exits.


John

[Alex Martelli]

Re: Python embedded in C &amp; memory releasing

John J. Lee wrote:
[color=blue]
> leonardbocock@y ahoo.com (lebo) writes:
>[color=green]
>> I'm trying to understand how Python handles memory usage and dynamic
>> object loading and unloading. Problem to solve? Build a very low
>> memory footprint (non-GUI) Python application for Unix/Windows. Why
>> use Python? End user flexibility.
>>
>> So far it seems that (on Windows):
>> (1) memory increases when you import <module>. Expected behaviour.
>> (2) memory does not release when you del <module>. Why not? Is Python
>> not releasing the memory or is it the OS not releasing memory?[/color]
> [...]
>
> On almost all OSes, processes hold on to all memory that has been
> allocated by the OS. Memory released by a process can only be reused
> *by that same process*. The OS only gets it back when the process
> exits.[/color]

This is not true any more these days!!! Consider the following program:

#include <stdio.h>
#include <stdlib.h>

int main()
{
char* blobbone;

printf("No blob yet, measure me now, then press enter... ");
while(getchar() != '\n') {}
printf("\n");

blobbone = malloc(1024*102 4);

printf("I have the blob, measure me now, then press enter... ");
while(getchar() != '\n') {}
printf("\n");

free(blobbone);

printf("No more blob, measure me now, then press enter... ");
while(getchar() != '\n') {}
printf("\n");

printf("Bye bye.\n");
return 0;
}

Using Ctrl-Z to suspend and Enter to continue, on Linux, we see:

[alex@lancelot sae]$ gcc mem.c
[alex@lancelot sae]$ ./a.out
No blob yet, measure me now, then press enter...
[1]+ Stopped ./a.out
[alex@lancelot sae]$ ps -C a.out v
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
32606 pts/6 T 0:00 73 1 1330 276 0.0 ./a.out
[alex@lancelot sae]$ fg
../a.out


I have the blob, measure me now, then press enter...
[1]+ Stopped ./a.out
[alex@lancelot sae]$ ps -C a.out v
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
32606 pts/6 T 0:00 82 1 2362 320 0.0 ./a.out
[alex@lancelot sae]$ fg
../a.out


No more blob, measure me now, then press enter...
[1]+ Stopped ./a.out
[alex@lancelot sae]$ ps -C a.out v
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
32606 pts/6 T 0:00 82 1 1334 316 0.0 ./a.out
[alex@lancelot sae]$ fg
../a.out


Bye bye.
[alex@lancelot sae]$

See? the DRS grows from 1330 to 2362 K when the blob is allocated,
then drops back to 1334 when the blob is freed again. I'm pretty
sure on Windows, with a good modern VC++ version, you'll see the
same, apart from the slightly higher difficulty of the "measure
me now" task;-).

And in Python...? *SAME THING*!!! Watch:

print "No blob yet, measure me now, then press enter...",
x = raw_input()
print

blobbone = 1024*256*[None]

print "I have the blob, measure me now, then press enter...",
x = raw_input()
print

del blobbone

print "No more blob, measure me now, then press enter...",
x = raw_input()
print

print "Bye bye."


[alex@lancelot sae]$ python mem.py
No blob yet, measure me now, then press enter...
[1]+ Stopped python mem.py
[alex@lancelot sae]$ ps -C python v
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
32648 pts/6 T 0:00 436 779 3864 2504 0.4 python mem.py
[alex@lancelot sae]$ fg
python mem.py


I have the blob, measure me now, then press enter...
[1]+ Stopped python mem.py
[alex@lancelot sae]$ ps -C python v
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
32648 pts/6 T 0:00 436 779 4892 3536 0.6 python mem.py
[alex@lancelot sae]$ fg
python mem.py


No more blob, measure me now, then press enter...
[1]+ Stopped python mem.py
[alex@lancelot sae]$ ps -C python v
PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
32648 pts/6 T 0:00 436 779 3864 2508 0.4 python mem.py
[alex@lancelot sae]$ fg
python mem.py


Bye bye.
[alex@lancelot sae]$

See? Just the same behavior -- DRS grows 3864->4892, then shrinks
back to 3864 when the blob is dropped.


The explanation of the behavior observed by the original poster is
quite different from the one you posit: rather, it is that
"del module" does NOT drop the module -- the module remains in
memory, fully available and loaded, and it is pointed to by
the appropriate entry sys.modules['module'] in case you want it
back. That's all there is to it -- modules, as opposed to, say,
lists, are NOT so easily disposed of:-).


Alex

[John J. Lee]

Re: Python embedded in C &amp; memory releasing

Alex Martelli <aleax@aleax.it > writes:[color=blue]
> John J. Lee wrote:[/color]
[...][color=blue][color=green]
> > On almost all OSes, processes hold on to all memory that has been
> > allocated by the OS. Memory released by a process can only be reused
> > *by that same process*. The OS only gets it back when the process
> > exits.[/color]
>
> This is not true any more these days!!! Consider the following program:[/color]
[...][color=blue]
> PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
> 32606 pts/6 T 0:00 73 1 1330 276 0.0 ./a.out[/color]
[...][color=blue]
> 32606 pts/6 T 0:00 82 1 2362 320 0.0 ./a.out[/color]
[...][color=blue]
> 32606 pts/6 T 0:00 82 1 1334 316 0.0 ./a.out[/color]
[...]

Well, you live and learn. :-)


John