Python loading library containing embedded python...

[Brennus]

I have a dll/so which embeds python. I can verify it works by
compiling it as an executable and adding an appropriate main.

I tried to write unit tests for this library with ctypes and a simple
python script. Access violations and other strange things result. I
suspect this is because I am basically embedding python in python at
this point.

How can I make my dll/so with embedded python support use via ctypes?

If Py_NewInterpret er is the answer, why does it hang indefinitely?

The dll/so must support use in processes which might contain other
instances of Python. I can not change that requirement. Running via
ctypes is an easy test of this capability (or much of it I suspect).

[Martin v. Löwis]

Re: Python loading library containing embedded python...

Brennus wrote:[color=blue]
> I have a dll/so which embeds python. I can verify it works by
> compiling it as an executable and adding an appropriate main.[/color]

Please explain in more detail how you did the embedding. Did you
statically link the Python interpreter into your dll/so, or did
you use a shared one?
[color=blue]
> How can I make my dll/so with embedded python support use via ctypes?[/color]

Whether you can do this at all depends on the fine details of
embedding: e.g. what operating system, how precisely did you do
the embedding, and so on.

For example, on many Unix systems, you might end up with two definitions
of all the Python symbols, at which point it depends on the precise
linker command line options to determine what precisely happens
(e.g. which of the definition of the symbol is used in what place).
[color=blue]
> The dll/so must support use in processes which might contain other
> instances of Python. I can not change that requirement. Running via
> ctypes is an easy test of this capability (or much of it I suspect).[/color]

Then your best bet is to *really* embed Python. I.e. link all symbols
statically, preferably renaming them (if possible), and make sure that
the resulting dll does neither reference nor expose any Python symbols.
Given the different linkers on different systems, this is very tricky
to implement.

Regards,
Martin

[Brennus]

Re: Python loading library containing embedded python...

Martin v. Löwis wrote:
[color=blue]
>Brennus wrote:[color=green]
>> I have a dll/so which embeds python. I can verify it works by
>> compiling it as an executable and adding an appropriate main.[/color]
>
>Please explain in more detail how you did the embedding. Did you
>statically link the Python interpreter into your dll/so, or did
>you use a shared one?[/color]

I compiled pythoncore.lib. I statically link this to my dll
and define Py_BUILD_CORE before including Python.h.

The only difference I can see with doing this vs linking the
python dll dynamically is that Py_IsInitialize d returns true
when using the dll. This makes sense, but little else does.
[color=blue][color=green]
>> How can I make my dll/so with embedded python support use via ctypes?[/color]
>
>Whether you can do this at all depends on the fine details of
>embedding: e.g. what operating system, how precisely did you do
>the embedding, and so on.
>
>For example, on many Unix systems, you might end up with two definitions
>of all the Python symbols, at which point it depends on the precise
>linker command line options to determine what precisely happens
>(e.g. which of the definition of the symbol is used in what place).
>[color=green]
>> The dll/so must support use in processes which might contain other
>> instances of Python. I can not change that requirement. Running via
>> ctypes is an easy test of this capability (or much of it I suspect).[/color]
>
>Then your best bet is to *really* embed Python. I.e. link all symbols
>statically, preferably renaming them (if possible), and make sure that
>the resulting dll does neither reference nor expose any Python symbols.
>Given the different linkers on different systems, this is very tricky
>to implement.[/color]

For now, I am testing this in Windows with Visual Studio.

I will look into renaming the symbols, but I am not sure
I understand why this is necessary. Surely two unrelated
DLLs can exist in a process with the same function names?
If not, I am surprised this problem hasn't been seen or
even exploited by plugin competitors more often.

Most of the time Py_Initialize and Py_IsInitialize d works.
High level tests with PyRun_SimpleStr ing("") crash in any
DLL I use from ctypes, while working fine as an executable.

[Brennus]

Re: Python loading library containing embedded python...

[color=blue]
>I have a dll/so which embeds python. I can verify it works by
>compiling it as an executable and adding an appropriate main.
>
>I tried to write unit tests for this library with ctypes and a simple
>python script. Access violations and other strange things result. I
>suspect this is because I am basically embedding python in python at
>this point.
>
>How can I make my dll/so with embedded python support use via ctypes?
>
>If Py_NewInterpret er is the answer, why does it hang indefinitely?
>
>The dll/so must support use in processes which might contain other
>instances of Python. I can not change that requirement. Running via
>ctypes is an easy test of this capability (or much of it I suspect).[/color]

I have solved my problem, or at least part of it by downgrading
to Visual Studio 2003. For whatever reason, the same steps applied
to Visual Studio 2005 result in massive problems.

Even the python.exe compiled by VS 2005 throws A/Vs on startup.

Does Python have to use Visual Studio? I have, of course, seen
the MinGW Python patches and tutorials, but I am told I should
not use anything but what Python was compiled with.

Additionally, since a Python compiled with even VS 2005 is majorly
broken, I am wary of using MinGW unless it is officially supported.

Are there plans to move away from VS? It's horrible imho. I only
upgraded from VS 6 at the recommendation I should use the same
compiler used to compile the latest Python. I asked why, citing my
own knowledge of struct alignment concerns between compilers, but
was vaguely told it was very bad, and that FILE structs can differ.

[Alex Martelli]

Re: Python loading library containing embedded python...

Brennus <xorchid@gmail. com> wrote:
...[color=blue]
> I have solved my problem, or at least part of it by downgrading
> to Visual Studio 2003. For whatever reason, the same steps applied
> to Visual Studio 2005 result in massive problems.
>
> Even the python.exe compiled by VS 2005 throws A/Vs on startup.[/color]

I believe the python core developers who do develop on Windows have not
even tried VS 2005 and have no intention of so doing for the foreseeable
future. Unfortunate -- as a python core developer myself, while I
mostly use MacOSX, and secondarily Linux, I'd be happy to lend a hand
occasionally with Windows *IF* I could use VS 2005, which I can get for
free, rather than any product I'd have to pay for.

But, it's not up to me: I use Windows only when I really truly have to
(e.g., to check that things work there, too, before publishing articles
or books) -- I WON'T commit to spending substantial time and effort on
Windows as long as I can use GOOD systems like the Mac or Linux (I used
to make a living as a Windows guru, see <http://www.aleax.it/TutWin32/>
for a trace of that, and exactly because of that I know use Windows as
rarely as I can get away with!-).

The choice of what Windows environment is best is obviously up to those
python core developers who DO use Windows day-in, day-out (such as Tim
Peters -- there may be others, but I can't really think of any offhand).
[color=blue]
> Are there plans to move away from VS? It's horrible imho. I only[/color]

I don't know of any such plans, sadly. You may want to take up the
subject on python-dev: few core developers routinely follow this list
(or newsgroup, whatever;-). If you volunteer to port the development
environment for Windows to X, and can show that the performance with
whatever compiler X offers is better than VS's, you may stand a chance.
Good luck... you'll need quite a bit;-).


Alex

[Martin v. Löwis]

Re: Python loading library containing embedded python...

Brennus wrote:[color=blue]
> I have solved my problem, or at least part of it by downgrading
> to Visual Studio 2003. For whatever reason, the same steps applied
> to Visual Studio 2005 result in massive problems.
>
> Even the python.exe compiled by VS 2005 throws A/Vs on startup.
>
> Does Python have to use Visual Studio?[/color]

Is this the debug build, by any chance? Microsoft broke support
for ISO C in VS2005. The Python trunk has a work-around for that
breakage.
[color=blue]
> I have, of course, seen
> the MinGW Python patches and tutorials, but I am told I should
> not use anything but what Python was compiled with.[/color]

More likely, you managed to mix msvcrt versions somehow. You
need to inspect your resulting DLL in depends.exe; please report
what it depends on.

Also notice that you cannot use extension DLLs in your embedded
Python: all extension DLLs depend on python2x.dll, so importing
one of them leads to multiple copies of the Python code in your
interpreter.
[color=blue]
> Are there plans to move away from VS?[/color]

Well, VS is the the "platform compiler", so no. Of course, if
you wanted that to happen, you could write a PEP, and explain
what alternative compiler (plus alternative build procedure)
should be used.
[color=blue]
> It's horrible imho. I only
> upgraded from VS 6 at the recommendation I should use the same
> compiler used to compile the latest Python. I asked why, citing my
> own knowledge of struct alignment concerns between compilers, but
> was vaguely told it was very bad, and that FILE structs can differ.[/color]

The problem really is with multiple copies of msvcrt. The struct
alignment in FILE structs is the same across all releases of
VC, yet having two copies of msvcrt in memory still might lead
to crashes.

The problem is that each msvcrt has its own FILE table, so
each one has its own stdout, for example. Now, for ages,
Microsoft also supports multi-threading in msvcrt, so there
was a need to extend struct FILE. They did so by introducing
struct FILEX, with an additional lock field. For
compatibility, they couldn't change the layout of the existing
array of FILE, so for the first 20 FILE values, the
lock is stored elsewhere (not in the __iob array, but in a
separate _locktable).

Now, when a file gets locked, msvcrt checks whether its pointer
is between __iob and __iob+20; if it is, it assumes it is
its own FILE, and looks for the lock in _locktable, else it
assumes it is FILEX, and looks for the lock in the struct itself.
If the FILE* really points into __iob of a different msvcrt,
that crashes.

Similar issues exist for malloc/free: you must not free a
pointer allocated by the malloc of a different msvcrt. Each
mscvrt has its own heap, and releasing memory to the wrong
heap might cause memory leaks.

Regards,
Martin

[Martin v. Löwis]

Re: Python loading library containing embedded python...

Brennus wrote:[color=blue]
> I compiled pythoncore.lib. I statically link this to my dll
> and define Py_BUILD_CORE before including Python.h.[/color]

Can you please elaborate? How did you do this? How do you know
pythoncore.lib is a static library? What msvcrt is it linked
with (use depends.exe to find out)?
[color=blue]
> The only difference I can see with doing this vs linking the
> python dll dynamically is that Py_IsInitialize d returns true
> when using the dll. This makes sense, but little else does.[/color]

You should explicitly call Py_Initialize in your embedded
application. Py_Initialize should return false before you
do so, and true afterwards. If that is not the case (e.g.
if Py_Initialize returns true even though you haven't called
Py_Initialize), something is wrong.
[color=blue]
> For now, I am testing this in Windows with Visual Studio.
>
> I will look into renaming the symbols, but I am not sure
> I understand why this is necessary. Surely two unrelated
> DLLs can exist in a process with the same function names?[/color]

On Windows, yes. Not on Unix, atleast not in general.
[color=blue]
> If not, I am surprised this problem hasn't been seen or
> even exploited by plugin competitors more often.[/color]

For plugins, there is a real problem on Unix, with several
strategies to solve it available. Avoiding naming conflicts
in the first place is the only one that really works,
independently of the plugin host.

Regards,
Martin

[Martin v. Löwis]

Re: Python loading library containing embedded python...

Alex Martelli wrote:[color=blue]
> I believe the python core developers who do develop on Windows have not
> even tried VS 2005 and have no intention of so doing for the foreseeable
> future.[/color]

That is not true. I have already tried it. Many others did, too, and
found that Microsoft managed to break ISO C compliance in VS 2005
(in order to make it "safe").
The Python trunk has a work-around for that problem.
[color=blue]
> Unfortunate -- as a python core developer myself, while I
> mostly use MacOSX, and secondarily Linux, I'd be happy to lend a hand
> occasionally with Windows *IF* I could use VS 2005, which I can get for
> free, rather than any product I'd have to pay for.[/color]

You can use VS2005 just fine. You have to convert the project files,
and then you are done.
[color=blue]
> The choice of what Windows environment is best is obviously up to those
> python core developers who DO use Windows day-in, day-out[/color]

Not really. Users formulate all kinds of demands, and we listen to them.
Name an arbitrary Windows compiler, and somebody will have proposed to
use that as the compiler for the official Windows binaries. Mention that
you are going to stick with the compiler you use, and somebody will
complain. Mention that you are going to change compilers, and somebody
will complain. You can't win.
[color=blue]
> I don't know of any such plans, sadly. You may want to take up the
> subject on python-dev: few core developers routinely follow this list
> (or newsgroup, whatever;-). If you volunteer to port the development
> environment for Windows to X, and can show that the performance with
> whatever compiler X offers is better than VS's, you may stand a chance.
> Good luck... you'll need quite a bit;-).[/color]

I personally don't care much about the performance of the compiler.
For my own development, I care about the tool integration, in particular
about the debugger (I like the MSVC debugger quite a lot). For the
Python project, I care about the compilers people want to use to
build extensions with.

For example, you can't really build COM-based extensions with cygwin,
so if the official compiler would be cygwin, somebody would have to
find out how to build PythonWin (in addition, PythonWin also has
a strong dependency on MSVCRT).

Regards,
Martin

[Brennus]

Re: Python loading library containing embedded python...

"Martin v. Löwis" <martin@v.loewi s.de> wrote:
[color=blue]
>Brennus wrote:[color=green]
>> I have solved my problem, or at least part of it by downgrading
>> to Visual Studio 2003. For whatever reason, the same steps applied
>> to Visual Studio 2005 result in massive problems.
>>
>> Even the python.exe compiled by VS 2005 throws A/Vs on startup.
>>
>> Does Python have to use Visual Studio?[/color]
>
>Is this the debug build, by any chance? Microsoft broke support
>for ISO C in VS2005. The Python trunk has a work-around for that
>breakage.[/color]

Good to hear, for my sanity at least.

I was actually thinking Microsoft was losing or breaking C on
purpose after using VS2005, and even VS2003 to some degree.

I think it's some sort of tactic. That's another reason I don't
want to continue using VS for Python.
[color=blue][color=green]
>> I have, of course, seen
>> the MinGW Python patches and tutorials, but I am told I should
>> not use anything but what Python was compiled with.[/color]
>
>More likely, you managed to mix msvcrt versions somehow. You
>need to inspect your resulting DLL in depends.exe; please report
>what it depends on.[/color]

I can't manage to compile anything with VS when Code Generation
is not set to the same type across projects. I guess I will try
the VS 2005 route again with this and the VS2005 patches in mind.

I am setting Code Generation the same across projects so I figured
that meant I was using the same C runtimes.
[color=blue]
>Also notice that you cannot use extension DLLs in your embedded
>Python: all extension DLLs depend on python2x.dll, so importing
>one of them leads to multiple copies of the Python code in your
>interpreter.
>[color=green]
>> Are there plans to move away from VS?[/color]
>
>Well, VS is the the "platform compiler", so no. Of course, if
>you wanted that to happen, you could write a PEP, and explain
>what alternative compiler (plus alternative build procedure)
>should be used.[/color]

I imagine MinGW's build process would be much easier.
I *think* I have to keep two Python trees just to *easily*
compile static and dynamic import lib files from Python VS.

Otherwise, it is quite easy to overwrite a 3 meg static
pythoncore.lib with the 200k dynamic/import version.