segfault in extension module

[Nathaniel Echols]

I've written a function in C to perform protein sequence alignment. This
works fine in a standalone C program. I've added the necessary packaging
to use it in Python; it returns three strings and an integer. However, as
soon as the function is complete, I get a segfault and the interpreter
dies.

If I run Python interactively, just calling the function causes a
segfault. If I'm running a script, I can actually print out the return
values (which are what I'd expect - so something's working) but as soon as
the script is done I get the segfault again. I can even call the function
twice, with different arguments - and it works both times. So it appears
that the problem is with tying up loose ends.

How do I determine what is going wrong? I do not get any problem like
this in the C version. I am not using free() anywhere - I will eventually
need to fix this, but I cannot find any place where I might be accessing
unavailable memory. (Adding in free() does not make any difference in the
module, for what it's worth, but I've had some issues with the C program
so I've left it out.)

[I've also tried using PyMem_Malloc instead, throughout the C code.
Doesn't help.]

thanks,
Nat
(please reply directly!)

[Martin v. Löwis]

Re: segfault in extension module

Nathaniel Echols <echols@OCF.Ber keley.EDU> writes:
[color=blue]
> How do I determine what is going wrong?[/color]

I recommend to run your code (or the interactive python) in a debugger.
For example, with gdb, you'd get

gdb /usr/bin/python
(gdb) run[color=blue][color=green][color=darkred]
>>> import foo[/color][/color][/color]
[program crashes]
(gdb) bt

The latter command will give a backtrace, which should tell you where
it crashes. If you don't get enough detail, make sure you compile your
module with debugging information. If you are on Windows, make then
sure that Python is compiled for debugging as well.

Regards,
Martin

[Miki Tebeka]

Re: segfault in extension module

Hello Nat,
[color=blue]
> I've written a function in C to perform protein sequence alignment. This
> works fine in a standalone C program. I've added the necessary packaging
> to use it in Python;[/color]
Which one? Are you using the C API?
[color=blue]
> it returns three strings and an integer. However, as
> soon as the function is complete, I get a segfault and the interpreter
> dies.[/color]
Looks like refcount problems, check out
http://www.python.org/doc/current/ext/refcounts.html .

IMO just avoid all this stuff and use SWIG/Boost.Python/Pyrex.

HTH.
Miki

[Nathaniel Echols]

Re: segfault in extension module

> > I've written a function in C to perform protein sequence alignment. This[color=blue][color=green]
> > works fine in a standalone C program. I've added the necessary packaging
> > to use it in Python;[/color]
> Which one? Are you using the C API?[/color]

Yup - I'm reading right out of the manual.
[color=blue]
> Looks like refcount problems, check out
> http://www.python.org/doc/current/ext/refcounts.html .[/color]

I read this before and couldn't figure out what it meant. This does seem
like it would relate, but I can't figure out what I'm doing incorrectly.
I just have one function which calls a pure C function and returns a
tuple of strings from it. I'm guessing I need to add a Py_INCREF()
somewhere but so far this just makes it segfault sooner. (I'm
not sure what argument to use for Py_INCREF(), either.)

I've looked at several other pages, and they all seem to involve setups
more complicated than what I'm doing. I'm already using Py_BuildValue()
to generate the returned tuple, and my understanding is that this should
avoid major problems. . .
[color=blue]
> IMO just avoid all this stuff and use SWIG/Boost.Python/Pyrex.[/color]

I'll look at these, but I only have a tiny little bit of code I need to do
this with - I coded it from scratch with the intention of using it this
way, and could have written it in Python if I didn't care about speed.
Would I really benefit from using one of the other methods? The goal here
is explicitly to put only the very time-dependent code in C; everything
else stays in Python.

thanks,
Nat

[Nathaniel Echols]

Re: segfault in extension module

> The latter command will give a backtrace, which should tell you where it[color=blue]
> crashes. If you don't get enough detail, make sure you compile your
> module with debugging information. If you are on Windows, make then sure
> that Python is compiled for debugging as well. Regards, Martin[/color]

Okay:

#0 0x420744fe in _int_free () from /lib/tls/libc.so.6
#1 0x420734d6 in free () from /lib/tls/libc.so.6
#2 0x0809dc0d in _PyObject_GC_De l ()
#3 0x080ce86f in PyDict_Next ()
#4 0x080d191d in _PyModule_Clear ()

I guess this makes sense, but I'm still not sure how to fix it. . .

[John J. Lee]

Re: segfault in extension module

Nathaniel Echols <echols@OCF.Ber keley.EDU> writes:
[...][color=blue][color=green]
> > IMO just avoid all this stuff and use SWIG/Boost.Python/Pyrex.[/color]
>
> I'll look at these, but I only have a tiny little bit of code I need to do[/color]
[...][color=blue]
> Would I really benefit from using one of the other methods?[/color]

Why not take advantage of it? You've already discovered how
hand-writing extensions can be painful. SWIG is probably best for
you. Ignore all the fancy SWIG features, just ask it to wrap the
function, it's very easy.

[color=blue]
> The goal here
> is explicitly to put only the very time-dependent code in C; everything
> else stays in Python.[/color]

SWIG should be able to do that. There's probably some overhead above
a hand-written wrapper, but it's so trivial to use SWIG in this simple
way that you probably shouldn't begin to worry about that -- it's
unlikely to be a problem.


John

[Michael Hudson]

Re: segfault in extension module

Nathaniel Echols <echols@OCF.Ber keley.EDU> writes:
[color=blue][color=green]
> > Looks like refcount problems, check out
> > http://www.python.org/doc/current/ext/refcounts.html .[/color]
>
> I read this before and couldn't figure out what it meant.[/color]

Then I am pretty sure this is your problem :-)
[color=blue]
> This does seem like it would relate, but I can't figure out what I'm
> doing incorrectly. I just have one function which calls a pure C
> function and returns a tuple of strings from it. I'm guessing I
> need to add a Py_INCREF() somewhere but so far this just makes it
> segfault sooner. (I'm not sure what argument to use for
> Py_INCREF(), either.)
>
> I've looked at several other pages, and they all seem to involve setups
> more complicated than what I'm doing. I'm already using Py_BuildValue()
> to generate the returned tuple, and my understanding is that this should
> avoid major problems. . .[/color]

Post some code.

Cheers,
mwh

--
If Unicode is a horde of zombies with flaming dung sticks,
the hideous intricacies of JIS, Chinese Big-5, Chinese
Traditional, KOI-8, et cetera are at least an army of ogres
with salt and flensing knives. -- Eric S. Raymond, python-dev

[Martin v. Löwis]

Re: segfault in extension module

Nathaniel Echols <echols@OCF.Ber keley.EDU> writes:
[color=blue]
> Okay:
>
> #0 0x420744fe in _int_free () from /lib/tls/libc.so.6
> #1 0x420734d6 in free () from /lib/tls/libc.so.6
> #2 0x0809dc0d in _PyObject_GC_De l ()
> #3 0x080ce86f in PyDict_Next ()
> #4 0x080d191d in _PyModule_Clear ()
>
> I guess this makes sense, but I'm still not sure how to fix it. . .[/color]

Ah, tls/libc.so.6. I think you lose, being confronted with a buggy C
library. Try making it not use /lib/tls.

If that does not change the behaviour, you probably have a
ref-counting bug somewhere. Try building a debugging version of
Python.

Regards,
Martin

[Nat Echols]

Re: segfault in extension module

> Post some code.

may god have mercy on my soul:

#include <Python.h>
#include "nw.h"


static PyObject *nw_align (PyObject *self, PyObject *args) {
char *seq1, *seq2, *mfile;
char *out1, *out2, *match;
int penalty, status, score;
PyObject *results;

if (! PyArg_ParseTupl e(args, "sssi", &seq1, &seq2, &mfile, &penalty)) {
return NULL;
}

status = nw(seq1, seq2, mfile, penalty, &out1, &out2, &match, &score);

if (status == -1) {
PyErr_NoMemory( );
return NULL;
}

results = Py_BuildValue(" (sssi)", out1, out2, match, score);
return results;
}

static PyMethodDef nwMethods[] = {
{"align", nw_align, METH_VARARGS,
"Perform Needleman-Wunsch alignment of two protein sequences."},
{NULL, NULL, 0, NULL} /* This is required! No idea why. */
};

void initnw (void) {
(void) Py_InitModule(" nw", nwMethods);
}

This is just the wrapper; the function that it calls is defined such:
int nw (char *seq1, char *seq2, char *matrixfile, int penalty,
char **_out1, char **_out2, char **_match, int *_score);
(I can supply this too, but I already know this works fine in a standalone
C program.)

In Python, I simply do this:
(out1, out2, match, score) = nw.align(seq1, seq2, "BLOSUM62", 0)

This is it; should be simple to fix, no?

[Michael Hudson]

Re: segfault in extension module

Nat Echols <echols@uclink. berkeley.edu> writes:
[color=blue][color=green]
> > Post some code.[/color]
>
> may god have mercy on my soul:[/color]

Well, I can't see anything flagrantly wrong with that.

Does a debugger provide any hints?

Cheers,
mwh

--
[3] Modem speeds being what they are, large .avi files were
generally downloaded to the shell server instead[4].
[4] Where they were usually found by the technical staff, and
burned to CD. -- Carlfish, asr

[Nat Echols]

Re: segfault in extension module

> Well, I can't see anything flagrantly wrong with that.[color=blue]
>
> Does a debugger provide any hints?[/color]

Someone else suggested this, and this is what the backtrace from gdb
indicated:

#0 0x420744fe in _int_free () from /lib/tls/libc.so.6
#1 0x420734d6 in free () from /lib/tls/libc.so.6
#2 0x0809dc0d in _PyObject_GC_De l ()
#3 0x080ce86f in PyDict_Next ()
#4 0x080d191d in _PyModule_Clear ()
(It goes on like this for a while. . .)

As far as the tls C library, which someone told me was buggy - I don't
know how to force use of a different library; Python appears to be linked
to the same one. At any rate, I seem to have the same problem on two
different machines, one running SuSE 8.1, one running RedHat 9.0.

thanks,
Nat

[Michael Hudson]

Re: segfault in extension module

Nat Echols <echols@uclink. berkeley.edu> writes:
[color=blue][color=green]
> > Well, I can't see anything flagrantly wrong with that.
> >
> > Does a debugger provide any hints?[/color]
>
> Someone else suggested this, and this is what the backtrace from gdb
> indicated:
>
> #0 0x420744fe in _int_free () from /lib/tls/libc.so.6
> #1 0x420734d6 in free () from /lib/tls/libc.so.6
> #2 0x0809dc0d in _PyObject_GC_De l ()
> #3 0x080ce86f in PyDict_Next ()
> #4 0x080d191d in _PyModule_Clear ()[/color]
^^^^^^^^^^^^^^^
*That's* very odd.
[color=blue]
> (It goes on like this for a while. . .)[/color]

Can we see a little more?

Cheers,
mwh

--
ZAPHOD: Listen three eyes, don't try to outwierd me, I get stranger
things than you free with my breakfast cereal.
-- The Hitch-Hikers Guide to the Galaxy, Episode 7

[Nat Echols]

Re: segfault in extension module

> *That's* very odd.[color=blue]
>
> Can we see a little more?[/color]

Yup:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 1074141248 (LWP 5500)]
0x420744b0 in _int_free () from /lib/tls/libc.so.6
(gdb) bt
#0 0x420744b0 in _int_free () from /lib/tls/libc.so.6
#1 0x420734d6 in free () from /lib/tls/libc.so.6
#2 0x08055a1f in _PyObject_Del ()
#3 0x080593c5 in PyString_AsEnco dedString ()
#4 0x0807db19 in _PyEval_SliceIn dex ()
#5 0x080c41c9 in PyFunction_SetC losure ()
#6 0x080cdf49 in PyDict_New ()
#7 0x080ce35e in PyDict_SetItem ()
#8 0x080d1799 in _PyModule_Clear ()
#9 0x0808e81a in PyImport_Cleanu p ()
#10 0x08096114 in Py_Finalize ()
#11 0x080539bf in Py_Main ()
#12 0x08053469 in main ()
#13 0x42015574 in __libc_start_ma in () from /lib/tls/libc.so.6

Now, if I run it interactively instead (still within gdb), I get this
backtrace when it segfaults after I try to exit Python:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 1074141248 (LWP 5502)]
0x420744b0 in _int_free () from /lib/tls/libc.so.6
(gdb) bt
#0 0x420744b0 in _int_free () from /lib/tls/libc.so.6
#1 0x420734d6 in free () from /lib/tls/libc.so.6
#2 0x08055a1f in _PyObject_Del ()
#3 0x080593c5 in PyString_AsEnco dedString ()
#4 0x080ce4d9 in PyDict_DelItem ()
#5 0x0805f1f2 in PyString_Fini ()
#6 0x08096153 in Py_Finalize ()
#7 0x080539bf in Py_Main ()
#8 0x08053469 in main ()
#9 0x42015574 in __libc_start_ma in () from /lib/tls/libc.so.6