Embedded python with threads

[nik]

hi,

I have a C++ app that loads a python module and calls a function from
it. The call is getting the contents of a list that's global in that module.

However, I'd like the python script to be also running a thread that
fills that global list, but can't figure out how to do it. Essentially
the code looks like;

in the C++ app;

////
Py_Initialize() ;

// PyImport_Import Module blocks until the myModule script has run
// through...
PyObject* module = PyImport_Import Module("myModul e");

PyObject* function = PyObject_GetAtt rString(module, "GetList");
while(1) {
PyObject* pyList = PyObject_CallFu nction(function , "");
// use the stuff in pyList
sleep(15);
}
////

and in the myModule.py;

####
import thread
import time
orderListLock = thread.allocate _lock()

pyList = []

def GetList():
global pyList
tmpList = []
orderListLock.a cquire()
tmpList = pyList
orderListLock.r elease()
return tmpList

def keepFillingList Thread():
global pyList
while 1:
orderListLock.a cquire()
pyList.append(s omestuff)
orderListLock.r elease()
time.sleep(5)

# the following statement happens when the C++ app imports the module,
# but the thread only lives as long the main python thread (which is
# over straight away on my linux PC)

thread.start_ne w_thread(keepFi llingListThread , ())

####

Has anyone any ideas?

thanks,
nik

[nik]

Re: Embedded python with threads

nik wrote:[color=blue]
> hi,
>
> I have a C++ app that loads a python module and calls a function from
> it. The call is getting the contents of a list that's global in that
> module.
>
> However, I'd like the python script to be also running a thread that
> fills that global list, but can't figure out how to do it. Essentially
> the code looks like;
>
> in the C++ app;
>
> ////
> Py_Initialize() ;
>
> // PyImport_Import Module blocks until the myModule script has run
> // through...
> PyObject* module = PyImport_Import Module("myModul e");
>
> PyObject* function = PyObject_GetAtt rString(module, "GetList");
> while(1) {
> PyObject* pyList = PyObject_CallFu nction(function , "");
> // use the stuff in pyList
> sleep(15);
> }
> ////
>
> and in the myModule.py;
>
> ####
> import thread
> import time
> orderListLock = thread.allocate _lock()
>
> pyList = []
>
> def GetList():
> global pyList
> tmpList = []
> orderListLock.a cquire()
> tmpList = pyList
> orderListLock.r elease()
> return tmpList
>
> def keepFillingList Thread():
> global pyList
> while 1:
> orderListLock.a cquire()
> pyList.append(s omestuff)
> orderListLock.r elease()
> time.sleep(5)
>
> # the following statement happens when the C++ app imports the module,
> # but the thread only lives as long the main python thread (which is
> # over straight away on my linux PC)
>
> thread.start_ne w_thread(keepFi llingListThread , ())
>
> ####
>
> Has anyone any ideas?
>
> thanks,
> nik[/color]

Hi again,

I think I'm one step further. I've put the
thread.start_ne w_thread(keepFi llingListThread , ()) command into its own
function, like

def startThreads():
thread.start_ne w_thread(keepFi llingListThread , ())
while 1:
pass

and then in the C++ part, I've used boost::thread to create a thread to
call startThreads like;

void myBoostThread:: operator () ()
{
PyObject* function = PyObject_GetAtt rString(m_cfg.m odule, "createThreads" );
PyObject* result = PyObject_CallFu nction(function , "");
}

// after the PyObject* module = PyImport_Import Module("myModul e"); line
from above:
myBoostThreadPa rams params;
params.module = module;
myBoostThread mythread(params );
boost::thread theThread(mythr ead);

this seems to work (at least, the python threads stay alive), but it
rapidly crashes out with

Fatal Python error: ceval: tstate mix-up
Aborted

as soon as I call the function GetList above.

Can anyone help me?

thanks

[Greg Chapman]

Re: Embedded python with threads

On Wed, 15 Sep 2004 18:05:54 +0200, nik <my.news.groups @noos.fr> wrote:
[color=blue]
>////
>Py_Initialize( );
>
>// PyImport_Import Module blocks until the myModule script has run
>// through...
>PyObject* module = PyImport_Import Module("myModul e");
>
>PyObject* function = PyObject_GetAtt rString(module, "GetList");
>while(1) {
> PyObject* pyList = PyObject_CallFu nction(function , "");
> // use the stuff in pyList
> sleep(15);
> }[/color]

I'm not sure I fully understand your problem, but in the loop above, you should
have:

Py_BEGIN_ALLOW_ THREADS
sleep(15);
Py_END_ALLOW_TH READS

This releases the Python GIL during the sleep call, allowing your list-filling
thread to run.
[color=blue]
>
># the following statement happens when the C++ app imports the module,
># but the thread only lives as long the main python thread (which is
># over straight away on my linux PC)
>[/color]

This is what I don't understand, since the Python main thread should be the
thread on which Py_Initialize was called; it should remain alive (from Python's
perspective) until Py_Finalize is called.

---
Greg Chapman

[nik]

Re: Embedded python with threads

Greg Chapman wrote:[color=blue]
> On Wed, 15 Sep 2004 18:05:54 +0200, nik <my.news.groups @noos.fr> wrote:
>
>[color=green]
>>////
>>Py_Initialize ();
>>
>>// PyImport_Import Module blocks until the myModule script has run
>>// through...
>>PyObject* module = PyImport_Import Module("myModul e");
>>
>>PyObject* function = PyObject_GetAtt rString(module, "GetList");
>>while(1) {
>> PyObject* pyList = PyObject_CallFu nction(function , "");
>> // use the stuff in pyList
>> sleep(15);
>> }[/color]
>
>
> I'm not sure I fully understand your problem, but in the loop above, you should
> have:
>
> Py_BEGIN_ALLOW_ THREADS
> sleep(15);
> Py_END_ALLOW_TH READS
>
> This releases the Python GIL during the sleep call, allowing your list-filling
> thread to run.
>
>[color=green]
>># the following statement happens when the C++ app imports the module,
>># but the thread only lives as long the main python thread (which is
>># over straight away on my linux PC)
>>[/color]
>
>
> This is what I don't understand, since the Python main thread should be the
> thread on which Py_Initialize was called; it should remain alive (from Python's
> perspective) until Py_Finalize is called.
>[/color]

I think I was making an assumption that since the list filling threads
didn't run that the import module was calling the create thread command,
then exiting (killing the children threads along with it) - I'm a novice
at threads with python, so I'm probably misunderstandin g the general
picture. However, maybe your suggestion of the Py_BEGIN_ALLOW_ THREADS
will make the difference. I'll try again as soon as I can (not until
next week now)...

many thanks for the reply,
nik
[color=blue]
> ---
> Greg Chapman
>[/color]

[Greg Chapman]

Re: Embedded python with threads

On Fri, 17 Sep 2004 01:05:13 +0200, nik <my.news.groups @noos.fr> wrote:
[color=blue]
>I think I was making an assumption that since the list filling threads
>didn't run that the import module was calling the create thread command,
>then exiting (killing the children threads along with it) - I'm a novice
>at threads with python, so I'm probably misunderstandin g the general
>picture. However, maybe your suggestion of the Py_BEGIN_ALLOW_ THREADS
>will make the difference. I'll try again as soon as I can (not until
>next week now)...[/color]

OK, this sounds like you were definitely not releasing the GIL (the Global
Interpreter Lock). To get an idea of how Python's threading works (from the C
side of things), I'd suggest reading this:

http://www.python.org/dev/doc/devel/api/threads.html

That's from the documentation for 2.4, but I believe everything there applies to
2.3 as well. I'm linking to it because it includes some discussion of the
PyGILState functions (this discussion is missing from the 2.3 docs, but the
functions are there).

---
Greg Chapman