Need some help with Python/C api and threading

[Steve Menard]

Here is my problem.

I have this library thats hosts another language within python, and
allows that language to call back INTO python.

All is good as long as the other languages calls back on the same
thread. If the callback arrives on a different thread, all hell break
loose and the program dies horribly.

looking at the C api documentation, I came upon the following block of
code :

PyThreadState *tstate;
PyObject *result;

/* interp is your reference to an interpreter object. */
tstate = PyThreadState_N ew(interp);
PyEval_AcquireT hread(tstate);

/* Perform Python actions here. */
result = CallSomeFunctio n();
/* evaluate result */

/* Release the thread. No Python API allowed beyond this point. */
PyEval_ReleaseT hread(tstate);

/* You can either delete the thread state, or save it
until you need it the next time. */
PyThreadState_D elete(tstate);


Which would seem to be what I need. However, I have no idea how to get
at that interp pointer. I tried the following :

PyInterpreterSt ate* interp = PyInterpreterSt ate_New();
PyThreadState *tstate = PyThreadState_N ew(interp);
PyEval_AcquireT hread(tstate);

but then it crashes on the second line ...

Anybody ever done this? As a side note, the hosted language can start an
arbitrary number of threads ...

Steve

[Les Smithson]

Re: Need some help with Python/C api and threading

>>>>> "Steve" == Steve Menard <steve.menard@v ideotron.ca> writes:

Steve> Here is my problem. I have this library thats hosts
Steve> another language within python, and allows that language to
Steve> call back INTO python.

Steve> All is good as long as the other languages calls back on
Steve> the same thread. If the callback arrives on a different
Steve> thread, all hell break loose and the program dies horribly.

Steve> looking at the C api documentation, I came upon the
Steve> following block of code :

Steve> PyThreadState *tstate; PyObject *result;

Steve> /* interp is your reference to an interpreter
Steve> object. */ tstate = PyThreadState_N ew(interp);
Steve> PyEval_AcquireT hread(tstate);

Steve> /* Perform Python actions here. */ result =
Steve> CallSomeFunctio n(); /* evaluate result */

Steve> /* Release the thread. No Python API allowed beyond
Steve> this point. */ PyEval_ReleaseT hread(tstate);

Steve> /* You can either delete the thread state, or save it
Steve> until you need it the next time. */
Steve> PyThreadState_D elete(tstate);


Steve> Which would seem to be what I need. However, I have no idea
Steve> how to get at that interp pointer. I tried the following :

Steve> PyInterpreterSt ate* interp =
Steve> PyInterpreterSt ate_New(); PyThreadState *tstate =
Steve> PyThreadState_N ew(interp); PyEval_AcquireT hread(tstate);

Steve> but then it crashes on the second line ...

Steve> Anybody ever done this? As a side note, the hosted language
Steve> can start an arbitrary number of threads ...

Steve> Steve

I haven't done this for a while and I'm a little hazy on it, so this
may be incorrect:

I used 'PyThreadState *ts = Py_NewInterpret er();' to set a new
sub-interpreter state if called in a new thread.

If the embedded script calls back into the extension, it restores that
thread state and acquires the GIL before making any other Py* calls by
calling 'PyEval_Restore Thread(ts);'. Before returning, it calls
'PyEval_SaveThr ead()'.

[Steve Menard]

Re: Need some help with Python/C api and threading

Les Smithson wrote:[color=blue][color=green][color=darkred]
>>>>>>"Steve" == Steve Menard <steve.menard@v ideotron.ca> writes:[/color][/color][/color]

<SNIP>
[color=blue]
>
> I haven't done this for a while and I'm a little hazy on it, so this
> may be incorrect:
>
> I used 'PyThreadState *ts = Py_NewInterpret er();' to set a new
> sub-interpreter state if called in a new thread.
>
> If the embedded script calls back into the extension, it restores that
> thread state and acquires the GIL before making any other Py* calls by
> calling 'PyEval_Restore Thread(ts);'. Before returning, it calls
> 'PyEval_SaveThr ead()'.
>
>[/color]

Thanks, however I dont think thid will work. The doc for
Py_NewInterpret er says that it created a "an (almost) totally separate
environment for the execution of Python code. In particular, the new
interpreter has separate, independent versions of all imported modules".
This is not good for me, as the callbacks must come in the "main"
interpreter context.

Is there a tutorial somewhere? Or a particularly well written extension
module whose source code I could take a look at?

Let me summarize my situation :

I am writing a python extension, not embedding python. As such, I have
no control over the interpreter, or the threads.

The library I am embedding is not of my own writing. It can create any
number of threads. It can make callbacks into the Python interpreter on
any such thread.

A given thread can original either in python or the library, but control
can go back and forth : A python method can call a library method, which
in turn calls back into python, which calls a linrary method, etc ...
This is a potential problem, because trying to grab in GIL twice from
the same thread will cause a deadlock.


So far, here is what I am doing (without success).

1) In the init_XXX method, I call PyEval_InitThre ads().

2) Every time I pass control to the library, I wrap the call into a
Py_BEGIN_ALLOW_ THREADS/Py_END_ALLOW_TH READS pair. Note that I adde this
recently, and get an error the second time Py_BEGIN_ALLOW_ THREADS is
called, with the following error "Fatal Python error: PyEval_SaveThre ad:
NULL tstate"

3) Finally, whenever I receive a callback from the library, I added
these lines to the start and end of the method :

PyInterpreterSt ate* interp = PyInterpreterSt ate_New();
PyThreadState *tstate = PyThreadState_N ew(interp);
PyEval_AcquireT hread(tstate);

and

PyEval_ReleaseT hread(tstate);
PyThreadState_D elete(tstate);
PyInterpreterSt ate_Delete(inte rp);


Thats about it. I am sure someone, somewhere has done what I need :(


Thanks for any help you can provide,

Steve

[Simon Dahlbacka]

Re: Need some help with Python/C api and threading

Steve Menard wrote:[color=blue]
> Les Smithson wrote:
>[color=green][color=darkred]
>>>>>>> "Steve" == Steve Menard <steve.menard@v ideotron.ca> writes:[/color][/color]
>
>
> <SNIP>
>[color=green]
>>
>> I haven't done this for a while and I'm a little hazy on it, so this
>> may be incorrect:
>>
>> I used 'PyThreadState *ts = Py_NewInterpret er();' to set a new
>> sub-interpreter state if called in a new thread.
>>
>> If the embedded script calls back into the extension, it restores that
>> thread state and acquires the GIL before making any other Py* calls by
>> calling 'PyEval_Restore Thread(ts);'. Before returning, it calls
>> 'PyEval_SaveThr ead()'.
>>
>>[/color]
>
> Thanks, however I dont think thid will work. The doc for
> Py_NewInterpret er says that it created a "an (almost) totally separate
> environment for the execution of Python code. In particular, the new
> interpreter has separate, independent versions of all imported modules".
> This is not good for me, as the callbacks must come in the "main"
> interpreter context.
>
> Is there a tutorial somewhere? Or a particularly well written extension
> module whose source code I could take a look at?
>
> Let me summarize my situation :
>
> I am writing a python extension, not embedding python. As such, I have
> no control over the interpreter, or the threads.
>
> The library I am embedding is not of my own writing. It can create any
> number of threads. It can make callbacks into the Python interpreter on
> any such thread.
>
> A given thread can original either in python or the library, but control
> can go back and forth : A python method can call a library method, which
> in turn calls back into python, which calls a linrary method, etc ...
> This is a potential problem, because trying to grab in GIL twice from
> the same thread will cause a deadlock.
>
>
> So far, here is what I am doing (without success).
>
> 1) In the init_XXX method, I call PyEval_InitThre ads().
>
> 2) Every time I pass control to the library, I wrap the call into a
> Py_BEGIN_ALLOW_ THREADS/Py_END_ALLOW_TH READS pair. Note that I adde this
> recently, and get an error the second time Py_BEGIN_ALLOW_ THREADS is
> called, with the following error "Fatal Python error: PyEval_SaveThre ad:
> NULL tstate"
>
> 3) Finally, whenever I receive a callback from the library, I added
> these lines to the start and end of the method :
>
> PyInterpreterSt ate* interp = PyInterpreterSt ate_New();
> PyThreadState *tstate = PyThreadState_N ew(interp);
> PyEval_AcquireT hread(tstate);
>
> and
>
> PyEval_ReleaseT hread(tstate);
> PyThreadState_D elete(tstate);
> PyInterpreterSt ate_Delete(inte rp);
>
>
> Thats about it. I am sure someone, somewhere has done what I need :(
>
>
> Thanks for any help you can provide,[/color]
would this:

http://www.python.org/peps/pep-0311.html

...be of any help?

/Simon

[Steve Menard]

Re: Need some help with Python/C api and threading

Simon Dahlbacka wrote:[color=blue]
> Steve Menard wrote:
>[color=green]
>> Les Smithson wrote:
>>[color=darkred]
>>>>>>>> "Steve" == Steve Menard <steve.menard@v ideotron.ca> writes:[/color][/color][/color]
[color=blue][color=green]
>>
>>
>> Thanks for any help you can provide,[/color]
>
> would this:
> http://www.python.org/peps/pep-0311.html
>
> ..be of any help?
>
> /Simon[/color]

YES!!!! thats exactly it!!! thanks!

Though I gotta ask, in which version of Python was this introduced?
It'll have a big influence over the compativility list.


Steve

[Simon Dahlbacka]

Re: Need some help with Python/C api and threading

> YES!!!! thats exactly it!!! thanks![color=blue]
>
> Though I gotta ask, in which version of Python was this introduced?
> It'll have a big influence over the compativility list.
>
>
> Steve[/color]

...looking through the cvs logs it seems to me it should be available
from 2.3beta1

(http://cvs.sourceforge.net/viewcvs.p...lude/pystate.h)

[Ronald Oussoren]

Re: Need some help with Python/C api and threading

If you're using Python 2.3 I'd use the API described in PEP 311:

http://www.python.org/peps/pep-0311.html

Ronald

On 16-jun-04, at 21:30, Steve Menard wrote:
[color=blue]
> Here is my problem.
>
> I have this library thats hosts another language within python, and
> allows that language to call back INTO python.
>
> All is good as long as the other languages calls back on the same
> thread. If the callback arrives on a different thread, all hell break
> loose and the program dies horribly.
>
> looking at the C api documentation, I came upon the following block of
> code :
>
> PyThreadState *tstate;
> PyObject *result;
>
> /* interp is your reference to an interpreter object. */
> tstate = PyThreadState_N ew(interp);
> PyEval_AcquireT hread(tstate);
>
> /* Perform Python actions here. */
> result = CallSomeFunctio n();
> /* evaluate result */
>
> /* Release the thread. No Python API allowed beyond this point. */
> PyEval_ReleaseT hread(tstate);
>
> /* You can either delete the thread state, or save it
> until you need it the next time. */
> PyThreadState_D elete(tstate);
>
>
> Which would seem to be what I need. However, I have no idea how to get
> at that interp pointer. I tried the following :
>
> PyInterpreterSt ate* interp = PyInterpreterSt ate_New();
> PyThreadState *tstate = PyThreadState_N ew(interp);
> PyEval_AcquireT hread(tstate);
>
> but then it crashes on the second line ...
>
> Anybody ever done this? As a side note, the hosted language can start
> an arbitrary number of threads ...
>
> Steve
> --
> http://mail.python.org/mailman/listinfo/python-list
>
>[/color]
--
X|support bv http://www.xsupport.nl/
T: +31 610271479 F: +31 204416173

[Dave Cole]

Re: Need some help with Python/C api and threading

Steve Menard wrote:[color=blue]
> Les Smithson wrote:
>[color=green][color=darkred]
>>>>>>> "Steve" == Steve Menard <steve.menard@v ideotron.ca> writes:[/color][/color]
>
>
> <SNIP>
>[color=green]
>>
>> I haven't done this for a while and I'm a little hazy on it, so this
>> may be incorrect:
>>
>> I used 'PyThreadState *ts = Py_NewInterpret er();' to set a new
>> sub-interpreter state if called in a new thread.
>>
>> If the embedded script calls back into the extension, it restores that
>> thread state and acquires the GIL before making any other Py* calls by
>> calling 'PyEval_Restore Thread(ts);'. Before returning, it calls
>> 'PyEval_SaveThr ead()'.
>>
>>[/color]
>
> Thanks, however I dont think thid will work. The doc for
> Py_NewInterpret er says that it created a "an (almost) totally separate
> environment for the execution of Python code. In particular, the new
> interpreter has separate, independent versions of all imported modules".
> This is not good for me, as the callbacks must come in the "main"
> interpreter context.
>
> Is there a tutorial somewhere? Or a particularly well written extension
> module whose source code I could take a look at?
>
> Let me summarize my situation :
>
> I am writing a python extension, not embedding python. As such, I have
> no control over the interpreter, or the threads.
>
> The library I am embedding is not of my own writing. It can create any
> number of threads. It can make callbacks into the Python interpreter on
> any such thread.
>
> A given thread can original either in python or the library, but control
> can go back and forth : A python method can call a library method, which
> in turn calls back into python, which calls a linrary method, etc ...
> This is a potential problem, because trying to grab in GIL twice from
> the same thread will cause a deadlock.
>
>
> So far, here is what I am doing (without success).
>
> 1) In the init_XXX method, I call PyEval_InitThre ads().
>
> 2) Every time I pass control to the library, I wrap the call into a
> Py_BEGIN_ALLOW_ THREADS/Py_END_ALLOW_TH READS pair. Note that I adde this
> recently, and get an error the second time Py_BEGIN_ALLOW_ THREADS is
> called, with the following error "Fatal Python error: PyEval_SaveThre ad:
> NULL tstate"
>
> 3) Finally, whenever I receive a callback from the library, I added
> these lines to the start and end of the method :
>
> PyInterpreterSt ate* interp = PyInterpreterSt ate_New();
> PyThreadState *tstate = PyThreadState_N ew(interp);
> PyEval_AcquireT hread(tstate);
>
> and
>
> PyEval_ReleaseT hread(tstate);
> PyThreadState_D elete(tstate);
> PyInterpreterSt ate_Delete(inte rp);
>
>
> Thats about it. I am sure someone, somewhere has done what I need :([/color]

In the code for the Sybase extension module I ended up with some code to
manage the GIL and a lock per database context and database connection.

http://www.object-craft.com.au/projects/sybase/download.html

The code is a little complicated in that you can disable all locking
support at compile time if it is not important to you.

- Dave

--

http://www.object-craft.com.au