embedding Python: how to avoid memory leaks?

[Andrew Trevorrow]

No response to my last message, so I'll try a different tack...

Does anyone know of, or even better, has anyone here written a
C++ application for Mac/Windows that allows users to run Python
scripts from within the app? Not just once, but many times in
a single session, and without leaking memory. Preferably an
open source app so I can see how it's done.

Our app (http://golly.sourceforge.net/) currently uses calls
like these every time a user decides to run a script:

Py_Initialize() ;
PyRun_SimpleStr ing("execfile(' foo.py')");
Py_Finalize();

But even if foo.py is *empty* the above calls result in a memory
leak of about 12K on Mac OS 10.3.9 (using Python 2.3) and about
11K on Windows 2000 (using Python 2.4.2). I wouldn't mind if
there was a one-off cost due to calling Py_Initialize the very
first time, but we see leaks every time a script is executed.

I've tried calling Py_Initialize just once (at app startup)
and Py_Finalize once on exit, but that doesn't really solve
anything. It avoids leaks when using trivial scripts (like
an empty .py file!) but we want to run some rather complicated
scripts that consume lots of memory, so we need a reliable way
to release that memory. I was surprised to discover that
Py_Finalize doesn't always do that.

Is there some magic Python code that can restore memory usage
to what it was before each execfile call? Something like
PostScript's save and restore.

I've been struggling with this problem for about a week now.
Having been seduced by Python's power and beauty I'd hate to
have to abandon it and switch to Perl or some other crappy
scripting language! Please help...

Andrew

[Torsten Bronger]

Re: embedding Python: how to avoid memory leaks?

Hallöchen!

Andrew Trevorrow <andrew@trevorr ow.com> writes:
[color=blue]
> [...]
>
> [...] Not just once, but many times in a single session, and
> without leaking memory. Preferably an open source app so I can
> see how it's done.
>
> Our app (http://golly.sourceforge.net/) currently uses calls
> like these every time a user decides to run a script:
>
> Py_Initialize() ;
> PyRun_SimpleStr ing("execfile(' foo.py')");[/color]

Does PyRun_AnyFile show the same effect? That's the way I'm about
to go.

Tschö,
Torsten.

--
Torsten Bronger, aquisgrana, europa vetus ICQ 264-296-646

[Andrew Trevorrow]

Re: embedding Python: how to avoid memory leaks?

> > Our app (http://golly.sourceforge.net/) currently uses calls[color=blue][color=green]
> > like these every time a user decides to run a script:
> >
> > Py_Initialize() ;
> > PyRun_SimpleStr ing("execfile(' foo.py')");[/color]
>
> Does PyRun_AnyFile show the same effect? That's the way I'm about
> to go.[/color]

I couldn't get the PyRun_*File* calls to work on Windows, presumably
because of the FILE* problem mentioned in the docs.

I'll be very surprised if it makes any difference to the memory
leak problem. Let me know how you get on!

Andrew

[Martin v. Löwis]

Re: embedding Python: how to avoid memory leaks?

Andrew Trevorrow wrote:[color=blue]
> Our app (http://golly.sourceforge.net/) currently uses calls
> like these every time a user decides to run a script:
>
> Py_Initialize() ;
> PyRun_SimpleStr ing("execfile(' foo.py')");
> Py_Finalize();
>
> But even if foo.py is *empty* the above calls result in a memory
> leak of about 12K on Mac OS 10.3.9 (using Python 2.3) and about
> 11K on Windows 2000 (using Python 2.4.2).[/color]

I could reproduce a memory leak with the code

#include <Python.h>
int main()
{
while(1){
Py_Initialize() ;
PyRun_SimpleStr ing("execfile(' foo.py')");
Py_Finalize();
}
}

However, I could not reproduce a memory leak with the code

#include <Python.h>
int main()
{
Py_Initialize() ;
while(1){
PyRun_SimpleStr ing("execfile(' foo.py')");
}
Py_Finalize();
}

So I recommend you do Py_Initialize only once. It is well-known
that initializing the Python interpreter allocates memory that
can never be freed, e.g. global variables in extension modules
(there just isn't any API to tell all the modules to release their
memory). So a cycle of Py_Initialize/Py_Finalize will certainly
leak.

OTOH, PyRun_SimpleStr ing shouldn't leak, and didn't when I
tried it.

Regards,
Martin

[Torsten Bronger]

Re: embedding Python: how to avoid memory leaks?

Hallöchen!

andrew@trevorro w.com (Andrew Trevorrow) writes:
[color=blue]
> [...]
>
> I couldn't get the PyRun_*File* calls to work on Windows, presumably
> because of the FILE* problem mentioned in the docs.[/color]

Which compiler do you use?

Tschö,
Torsten.

--
Torsten Bronger, aquisgrana, europa vetus ICQ 264-296-646

[Andrew Trevorrow]

Re: embedding Python: how to avoid memory leaks?

Torsten Bronger <bronger@physik .rwth-aachen.de> wrote:
[color=blue]
> andrew@trevorro w.com (Andrew Trevorrow) writes:
>[color=green]
> > [...]
> >
> > I couldn't get the PyRun_*File* calls to work on Windows, presumably
> > because of the FILE* problem mentioned in the docs.[/color]
>
> Which compiler do you use?[/color]

MSVC++ (version 6 from memory -- I do most of my development on the
Mac and fire up Virtual PC occasionally to test Win builds).

Andrew

[Andrew Trevorrow]

Re: embedding Python: how to avoid memory leaks?

martin@v.loewis .de wrote:
[color=blue]
> I could reproduce a memory leak with the code
>
> #include <Python.h>
> int main()
> {
> while(1){
> Py_Initialize() ;
> PyRun_SimpleStr ing("execfile(' foo.py')");
> Py_Finalize();
> }
> }
>
> However, I could not reproduce a memory leak with the code
>
> #include <Python.h>
> int main()
> {
> Py_Initialize() ;
> while(1){
> PyRun_SimpleStr ing("execfile(' foo.py')");
> }
> Py_Finalize();
> }
>
> So I recommend you do Py_Initialize only once. It is well-known
> that initializing the Python interpreter allocates memory that
> can never be freed, e.g. global variables in extension modules
> (there just isn't any API to tell all the modules to release their
> memory). So a cycle of Py_Initialize/Py_Finalize will certainly
> leak.[/color]

Surely that's a bug that should be fixed. There should be some way
to tell Python "release all the memory you've ever allocated and
start again with a clean slate".
[color=blue]
> OTOH, PyRun_SimpleStr ing shouldn't leak, and didn't when I
> tried it.[/color]

Ok, after reading http://evanjones.ca/python-memory.html I think I
understand what's happening. Apparently the Python memory allocator
never releases memory back to the OS! So if a complicated script
happens to consume 100MB of interpreter memory then that amount is
no longer available to the app in which Python is embededded.
Even worse, if a script has a (Python) memory leak then there's
nothing the app can do about it. It would be great if wrapping
each script inside Py_Initialize/Py_Finalize could avoid all that.

I've been told that the next version of Python will release memory,
so that's good news. You can get it now if you're willing to build
Python from the latest source code.

Andrew

[Torsten Bronger]

Re: embedding Python: how to avoid memory leaks?

Hallöchen!

andrew@trevorro w.com (Andrew Trevorrow) writes:
[color=blue]
> Torsten Bronger <bronger@physik .rwth-aachen.de> wrote:
>[color=green]
>> andrew@trevorro w.com (Andrew Trevorrow) writes:
>>[color=darkred]
>>> [...]
>>>
>>> I couldn't get the PyRun_*File* calls to work on Windows,
>>> presumably because of the FILE* problem mentioned in the docs.[/color]
>>
>> Which compiler do you use?[/color]
>
> MSVC++ (version 6 from memory -- I do most of my development on
> the Mac and fire up Virtual PC occasionally to test Win builds).[/color]

Well, I don't really *know*, but it's hard to believe to me that the
file descriptor format changed within the Microsoft product series.

Tschö,
Torsten.

--
Torsten Bronger, aquisgrana, europa vetus ICQ 264-296-646

[Martin v. Löwis]

Re: embedding Python: how to avoid memory leaks?

Andrew Trevorrow wrote:[color=blue]
> Surely that's a bug that should be fixed. There should be some way
> to tell Python "release all the memory you've ever allocated and
> start again with a clean slate".[/color]

This bug cannot be fixed in any foreseeable future.
[color=blue]
> I've been told that the next version of Python will release memory,
> so that's good news. You can get it now if you're willing to build
> Python from the latest source code.[/color]

That still won't release all memory - only the arenas that don't
have live Python objects on them anymore.

Regards,
Martin

[Martin v. Löwis]

Re: embedding Python: how to avoid memory leaks?

Torsten Bronger wrote:[color=blue][color=green][color=darkred]
>>>>I couldn't get the PyRun_*File* calls to work on Windows,
>>>>presumabl y because of the FILE* problem mentioned in the docs.[/color][/color][/color]
[color=blue]
> Well, I don't really *know*, but it's hard to believe to me that the
> file descriptor format changed within the Microsoft product series.[/color]

The layout of the FILE type indeed didn't change. However, passing
FILE* across different versions of msvcrt will still crash; google
for details. In particular, if you build an application with VC6,
it will be linked with msvcrt4.dll. If you combine this with Python
2.4 (which is linked with msvcr71.dll), you cannot use PyRun_*File*.

Regards,
Martin

[Wolfgang]

Re: embedding Python: how to avoid memory leaks?

Hallo,
[color=blue][color=green][color=darkred]
> >>> I couldn't get the PyRun_*File* calls to work on Windows,
> >>> presumably because of the FILE* problem mentioned in the docs.
> >>
> >> Which compiler do you use?[/color]
> >
> > MSVC++ (version 6 from memory -- I do most of my development on
> > the Mac and fire up Virtual PC occasionally to test Win builds).[/color]
>
> Well, I don't really *know*, but it's hard to believe to me that the
> file descriptor format changed within the Microsoft product series.[/color]

Yes it is hard to believe but true.
Even debug and release versions have incompatible file pointer.
It's not possible to pass af FILE* from a debug app to the embedded
release python.dll.

A workaround is to create the FILE* with the python api functions and
pass this pointer back to python.