segfault when calling Python from C thread

[Travis Berg]

I'm running into a problem when trying to perform a callback to a Python
function from a C extension. Specifically, the callback is being made by
a pthread that seems to cause the problem. If I call the callback from
the parent process, it works fine. The PyObject is static, and holds the
same value in both Parent and thread, so I'm at a loss as to what would be
different in the pthread from the parent that would cause a segfault on
the callback. The machine specifics are an x86 intel processor with
RedHat linux.

Here is some clips of the C callback code showing how I'm storing the
callback, then what the actual callback function is like. Any ideas?
The function being called is a simply to display the string of text, and
execution never seems to reach back to the Python code at all.

Thanks,
Travis B.


/* callback function to the Python code */
static PyObject * my_callback = NULL;

/* setting callback function */
static PyObject * my_set_callback (PyObject *dummy, PyObject *args)
{
PyObject *result = NULL;
PyObject *temp;
PyObject *arglist;

if (PyArg_ParseTup le(args, "O:set_callback ", &temp)) {
if (!PyCallable_Ch eck(temp)) {
PyErr_SetString (PyExc_TypeErro r,
"parameter must be callable");
return NULL;
}
Py_XINCREF(temp ); /* Add a reference to new callback */
Py_XDECREF(my_c allback); /* Dispose of previous callback */
my_callback = temp; /* Remember new callback */
/* return "None" */
Py_INCREF(Py_No ne);
result = Py_None;
}
return result;
}

/* calling callback */
void callback(char * str) {
PyObject *arglist;
PyObject *result;
if(str == NULL)
return;

if(my_callback == NULL) {
printf("no callback function provided, returning...\n" );
return;
}

/* Time to call the callback */
arglist = Py_BuildValue(" (s)", str);
result = PyEval_CallObje ct(my_callback, arglist);
Py_DECREF(argli st);
if(result == NULL)
return;
Py_DECREF(resul t);
}

[Greg Chapman]

Re: segfault when calling Python from C thread

Travis Berg wrote:
[color=blue]
>
> I'm running into a problem when trying to perform a callback to a
> Python function from a C extension. Specifically, the callback is
> being made by a pthread that seems to cause the problem. If I call
> the callback from the parent process, it works fine. The PyObject is
> static, and holds the same value in both Parent and thread, so I'm at
> a loss as to what would be different in the pthread from the parent
> that would cause a segfault on the callback. The machine specifics
> are an x86 intel processor with RedHat linux.
>
>
> /* calling callback */
> void callback(char * str) {
> PyObject *arglist;
> PyObject *result;
> if(str == NULL)
> return;
>
> if(my_callback == NULL) {
> printf("no callback function provided, returning...\n" );
> return;
> }
>
> /* Time to call the callback */
> arglist = Py_BuildValue(" (s)", str);
> result = PyEval_CallObje ct(my_callback, arglist);
> Py_DECREF(argli st);
> if(result == NULL)
> return;
> Py_DECREF(resul t);
> }[/color]

Your callback function needs to hold the Python GIL (and have a vaild
threadstate) before it calls any Python C-API functions. Change the
last part of it to:

PyGILState_STAT E state;

/* ... */

/* Time to call the callback */

state = PyGILState_Ensu re();

arglist = Py_BuildValue(" (s)", str);
result = PyEval_CallObje ct(my_callback, arglist);
Py_DECREF(argli st);
if(result == NULL)
return;
Py_DECREF(resul t);

PyGILState_Rele ase(state);
}

Also, somewhere in your main thread you should call PyEval_InitThre ads
before any of the callback threads execute. (This call is made
automatically if you are creating new threads using Python's thread
module, but if the new threads are created by some C code, you need to
call it yourself.)

---
Greg Chapman

[Fredrik Lundh]

Re: segfault when calling Python from C thread

Greg Chapman wrote:
[color=blue]
> Your callback function needs to hold the Python GIL (and have a vaild
> threadstate) before it calls any Python C-API functions. Change the
> last part of it to:
>
> PyGILState_STAT E state;
>
> /* ... */
>
> /* Time to call the callback */
>
> state = PyGILState_Ensu re();
>
> arglist = Py_BuildValue(" (s)", str);
> result = PyEval_CallObje ct(my_callback, arglist);
> Py_DECREF(argli st);
> if(result == NULL)
> return;
> Py_DECREF(resul t);
>
> PyGILState_Rele ase(state);
> }[/color]

you might wish to make sure you release the GIL even if the callback
raises an exception...

</F>

[Greg Chapman]

Re: segfault when calling Python from C thread

Fredrik Lundh wrote:
[color=blue]
> Greg Chapman wrote:
>[color=green]
> > Your callback function needs to hold the Python GIL (and have a
> > vaild threadstate) before it calls any Python C-API functions.
> > Change the last part of it to:
> >
> > PyGILState_STAT E state;
> >
> > /* ... */
> >
> > /* Time to call the callback */
> >
> > state = PyGILState_Ensu re();
> >
> > arglist = Py_BuildValue(" (s)", str);
> > result = PyEval_CallObje ct(my_callback, arglist);
> > Py_DECREF(argli st);
> > if(result == NULL)
> > return;
> > Py_DECREF(resul t);
> >
> > PyGILState_Rele ase(state);
> > }[/color]
>
> you might wish to make sure you release the GIL even if the callback
> raises an exception...
>
> </F>[/color]

Argh, thanks for catching that. You probably put that too politely
though (in case anyone sees this who might think that is optional): one
should absolutely make sure all code paths which call PyGILState_Ensu re
have a matching call to PyGILState_Rele ase.

---
Greg Chapman