PyThreadState_Swap difference in 2.3.2?

[Paul Miller]

I have a C++ application that uses multiple embedded Python interpreters. I
allocate multiple interpreters using Py_NewInterpret er, and switch between
them using PyThreadSate_Sw ap.

In 2.2.2, this all worked fine.

I just installed 2.3.2, but now the first time I call Py_NewInterpret er, it
bails out in PyThreadState_S wap, in the debug checking code, with:

Py_FatalError(" Invalid thread state for this thread");

Has the interpreter/thread setup code changed since 2.2?

To be complete, I set up with this:

Py_Initialize() ;
PyThreadState *globalState = PyThreadState_G et();

// allocate new interpreter
PyThreadState *interp = Py_NewInterpret er();

I am investigating to see if a crash I used to see with 2.2 is still around
in 2.3, where if you call Py_Initialize() and Py_Finalize() multiple times
it would die.

[vincent wehren]

Re: PyThreadState_S wap difference in 2.3.2?

"Paul Miller" <paul@fxtech.co m> schrieb im Newsbeitrag
news:knbstvgsn3 o3qmmtu975g8eb9 4rhpmae2o@4ax.c om...
| I have a C++ application that uses multiple embedded Python interpreters.
I
| allocate multiple interpreters using Py_NewInterpret er, and switch between
| them using PyThreadSate_Sw ap.
|
| In 2.2.2, this all worked fine.
|
| I just installed 2.3.2, but now the first time I call Py_NewInterpret er,
it
| bails out in PyThreadState_S wap, in the debug checking code, with:
|
| Py_FatalError(" Invalid thread state for this thread");
|
| Has the interpreter/thread setup code changed since 2.2?
|
| To be complete, I set up with this:
|
| Py_Initialize() ;
| PyThreadState *globalState = PyThreadState_G et();
|
| // allocate new interpreter
| PyThreadState *interp = Py_NewInterpret er();

The same thing stumped me too a couple of days ago (s.
http://groups.google.nl/groups?q=PyN...home.nl&rnum=1 )
with 2.3.3c1 (same applies 2.4a0 for that matter).

Although I am sure the following snip in PyThreadState_S wap was put there
for a reason,
it would be great if someone could put some light on the matter....:

/* It should not be possible for more than one thread state
to be used for a thread. Check this the best we can in debug
builds.
*/
#if defined(Py_DEBU G) && defined(WITH_TH READ)
if (new) {
PyThreadState *check = PyGILState_GetT hisThreadState( );
if (check && check != new)
Py_FatalError(" Invalid thread state for this thread");
}
#endif

Regards
Vincent Wehren


|
| I am investigating to see if a crash I used to see with 2.2 is still
around
| in 2.3, where if you call Py_Initialize() and Py_Finalize() multiple times
| it would die.
|
|

[Paul Miller]

Re: PyThreadState_S wap difference in 2.3.2?

>The same thing stumped me too a couple of days ago (s.[color=blue]
>http://groups.google.nl/groups?q=PyN...home.nl&rnum=1 )
>with 2.3.3c1 (same applies 2.4a0 for that matter).[/color]

Interesting. Unfortunately, there isn't a whole lot of information out
there on what we're trying to do.

I went ahead and commented out that check, but then I came across another
problem. In object.c, it hit an assertion:

assert(mro != NULL);

This was for a custom Python type I had created in C. It was assuming this
mro field was initialized, which it wasn't. It must be something new,
because this all used to work in 2.2

I will have to keep looking at the release notes.

I wish there was a "upgrading from 2.2 to 2.3" document for people who are
writing custom Python extensions, and doing embedding and such.

[Mark Hammond]

Re: PyThreadState_S wap difference in 2.3.2?

vincent wehren wrote:[color=blue]
> "Paul Miller" <paul@fxtech.co m> schrieb im Newsbeitrag
> news:knbstvgsn3 o3qmmtu975g8eb9 4rhpmae2o@4ax.c om...
> | I have a C++ application that uses multiple embedded Python interpreters.
> I
> | allocate multiple interpreters using Py_NewInterpret er, and switch between
> | them using PyThreadSate_Sw ap.
> |
> | In 2.2.2, this all worked fine.
> |
> | I just installed 2.3.2, but now the first time I call Py_NewInterpret er,
> it
> | bails out in PyThreadState_S wap, in the debug checking code, with:
> |
> | Py_FatalError(" Invalid thread state for this thread");
> |
> | Has the interpreter/thread setup code changed since 2.2?
> |
> | To be complete, I set up with this:
> |
> | Py_Initialize() ;
> | PyThreadState *globalState = PyThreadState_G et();
> |
> | // allocate new interpreter
> | PyThreadState *interp = Py_NewInterpret er();
>
> The same thing stumped me too a couple of days ago (s.
> http://groups.google.nl/groups?q=PyN...home.nl&rnum=1 )
> with 2.3.3c1 (same applies 2.4a0 for that matter).
>
> Although I am sure the following snip in PyThreadState_S wap was put there
> for a reason,
> it would be great if someone could put some light on the matter....:
>
> /* It should not be possible for more than one thread state
> to be used for a thread. Check this the best we can in debug
> builds.
> */
> #if defined(Py_DEBU G) && defined(WITH_TH READ)
> if (new) {
> PyThreadState *check = PyGILState_GetT hisThreadState( );
> if (check && check != new)
> Py_FatalError(" Invalid thread state for this thread");
> }
> #endif[/color]

Python stores information in the thread-state structure which is
specific to the thread - information such as the exception being
handled, the current recursion depth, etc.

The assertion above is reporting the fact that 2 distinct ThreadState
objects are trying to be used on a single thread. The same thread for
which you are trying to create a thread-state already has such a state.

Earlier versions of Python made no attempt to detect this, but as far as
everyone can tell, was always an implicit assumption.

Unfortunately, much of this stuff has never been thought through
correctly. Trying to work with multiple InterpreterStat e objects is
also very difficult, and in some cases simply does not work, as not all
global variables are stored in an InterpreterStat e. Theoretically
though, this is probably what you want - if a different InterpreterStat e
is current, I would expect a ThreadState specific to the
InterpreterStat e could be used - but I doubt it works <wink>

The easy answer is to stop trying to create multiple interpreter states,
then use the PyGILState calls to manage your thread-state. These should
be 2 lines per function (rather than the many that exist now). If you
need to build in both pre 2.3 and 2.3+:
#if (PY_VERSION_HEX >= 0x02030000)
#define USE_GILSTATE
#endif
/* Then */
#ifdef USE_GILSTATE
PyGILState_Stat e state = PyGILState_Ensu re();
#else
20 lines of existing code
#endif
// your body, and at the end
#ifdef USE_GILSTATE
PyGILState_Rele ase(state);
#else
existing code
#endif

Mark.

[Mark Hammond]

Re: PyThreadState_S wap difference in 2.3.2?

Paul Miller wrote:
[color=blue][color=green]
>>The same thing stumped me too a couple of days ago (s.
>>http://groups.google.nl/groups?q=PyN...home.nl&rnum=1 )
>>with 2.3.3c1 (same applies 2.4a0 for that matter).[/color]
>
>
> Interesting. Unfortunately, there isn't a whole lot of information out
> there on what we're trying to do.
>
> I went ahead and commented out that check, but then I came across another
> problem. In object.c, it hit an assertion:
>
> assert(mro != NULL);
>
> This was for a custom Python type I had created in C. It was assuming this
> mro field was initialized, which it wasn't. It must be something new,
> because this all used to work in 2.2
>
> I will have to keep looking at the release notes.
>
> I wish there was a "upgrading from 2.2 to 2.3" document for people who are
> writing custom Python extensions, and doing embedding and such.[/color]

This assertion appears in PyObject_Generi cGetAttr(). What was the
attribute being requested? I'm guessing your extension type is doing
something funky, as otherwise PyObject_Generi cGetAttr() would not be called.

Almost all of my types upgraded without any pain. A few of the win32com
ones did require work to work with tp_iter, but that is only as these
types are themselves doing funky things.

I *think* that having tp_flags as zero works OK, but as soon as you use
Py_TPFLAGS_DEFA ULT, you are expected to initialize your type
differently. Sorry, but I can recall any more details.

Mark.

[Paul Miller]

Re: PyThreadState_S wap difference in 2.3.2?

>Unfortunatel y, much of this stuff has never been thought through[color=blue]
>correctly. Trying to work with multiple InterpreterStat e objects is
>also very difficult, and in some cases simply does not work, as not all
>global variables are stored in an InterpreterStat e. Theoretically
>though, this is probably what you want - if a different InterpreterStat e
>is current, I would expect a ThreadState specific to the
>InterpreterSta te could be used - but I doubt it works <wink>[/color]

It used to work perfectly in 2.2. Thus my dilemma.
[color=blue]
>The easy answer is to stop trying to create multiple interpreter states,
>then use the PyGILState calls to manage your thread-state. These should
>be 2 lines per function (rather than the many that exist now). If you
>need to build in both pre 2.3 and 2.3+:[/color]

But, I'm not looking for a replacement for per-thread state. I am actually
maintaining multiple *interpreters*, each with all those global variables
you talked about. I need to do this because I want to be able to create an
"environmen t", load some scripts, run them, then delete that "environmen t"
without affecting other "environmen ts".

If this PyGILState actually is an entire Python interpreter environment,
then I'll happily switch to it.

[Paul Miller]

Re: PyThreadState_S wap difference in 2.3.2?

>> assert(mro != NULL);[color=blue][color=green]
>>
>> This was for a custom Python type I had created in C. It was assuming this
>> mro field was initialized, which it wasn't. It must be something new,
>> because this all used to work in 2.2[/color][/color]
[color=blue]
>This assertion appears in PyObject_Generi cGetAttr(). What was the
>attribute being requested? I'm guessing your extension type is doing
>something funky, as otherwise PyObject_Generi cGetAttr() would not be called.[/color]

I am using GenericGetAttr( ) so my tp_getset list will work. This used to
work in 2.2.
[color=blue]
>I *think* that having tp_flags as zero works OK, but as soon as you use
>Py_TPFLAGS_DEF AULT, you are expected to initialize your type
>differently. Sorry, but I can recall any more details.[/color]

Nope, that didn't help. Since I am using new-style types, I had to have
most of those flags set anyway.

It seems to me that GenericGetAttr( ) should not assume mro to be
initialized. I haven't come across any sample custom types that initialize
it, and for whatever reason, it is not getting set up automatically.

[Paul Miller]

Re: PyThreadState_S wap difference in 2.3.2?

>It seems to me that GenericGetAttr( ) should not assume mro to be[color=blue]
>initialized. I haven't come across any sample custom types that initialize
>it, and for whatever reason, it is not getting set up automatically.[/color]

I traced through my initialization code, where I am "readying" my custom
types, and it is failing in the mro_implementat ion() function in
typeobject.c. For whatever reason, the PySequence_List () call in this
function is failing.

I have no clue why at this point.