Are there memory limits for external C modules?

[david@quanterium.zzn.com]

I've been working on an external C module for Python in order to use
some of the functionality from Ethereal. Right now I'm getting
segfaults originating from within the Ethereal source code, but the
same code works fine when used normally (i.e. through Ethereal or
Tethereal). I'm wondering if there's some sort of memory limitation the
Python interpreter imposes on the C code it calls that I might be
running into, and if so, any ways to increase it?

Thanks,

David

[Fredrik Lundh]

Re: Are there memory limits for external C modules?

david@quanteriu m.zzn.com wrote:
[color=blue]
> I've been working on an external C module for Python in order to use
> some of the functionality from Ethereal. Right now I'm getting
> segfaults originating from within the Ethereal source code, but the
> same code works fine when used normally (i.e. through Ethereal or
> Tethereal). I'm wondering if there's some sort of memory limitation the
> Python interpreter imposes on the C code it calls that I might be
> running into[/color]

no.

I'd suspect, in order:

you've haven't initialized/configured the Ethereal library correctly
you're using the Ethereal API in the wrong way
you're accidentally overwriting some memory area
you're messing up the reference count for some python object
you're using the Python API in the wrong way
some other bug in your code
a bug in the part of the ethereal API you're using
a bug in your code
a bug in the Python API you're using
a bug in your code

</F>

[david@quanterium.zzn.com]

Re: Are there memory limits for external C modules?

Fredrik Lundh wrote:[color=blue]
> david@quanteriu m.zzn.com wrote:
>[color=green]
> > I've been working on an external C module for Python in order to use
> > some of the functionality from Ethereal. Right now I'm getting
> > segfaults originating from within the Ethereal source code, but the
> > same code works fine when used normally (i.e. through Ethereal or
> > Tethereal). I'm wondering if there's some sort of memory limitation the
> > Python interpreter imposes on the C code it calls that I might be
> > running into[/color]
>
> no.[/color]

Well, that answers that. I know memory limits can be an issue with the
Java VM, at least it was back when I was taking Java classes in college
and using version 1.1. So I thought there might be something similar
going on with Python, but I guess not.
[color=blue]
> I'd suspect, in order:
>
> you've haven't initialized/configured the Ethereal library correctly
> you're using the Ethereal API in the wrong way
> you're accidentally overwriting some memory area
> you're messing up the reference count for some python object
> you're using the Python API in the wrong way
> some other bug in your code
> a bug in the part of the ethereal API you're using
> a bug in your code
> a bug in the Python API you're using
> a bug in your code[/color]

I was able to modify my C code so that instead of being a Python
module, it runs as a standalone binary, and it works as it should.
Calling it with os.spawn* works. The two versions are essentially the
same, the primary differences being the necessary difference in how
arguments and return values are handled.

This will work if necessary, but I would think having it as a Python
module would be slightly more elegant and efficient since we avoid the
overhead of setting up new processes.

In case it helps anyone, I'm running Python 2.4.1 on Fedora Core 4, and
working with Ethereal 0.10.14. I'm using the Tethereal source code as
a guide to how to call into the Ethereal code.

- David

[Marc 'BlackJack' Rintsch]

Re: Are there memory limits for external C modules?

In <1138316171.607 603.299130@g49g 2000cwa.googleg roups.com>,
david@quanteriu m.zzn.com wrote:
[color=blue]
> I was able to modify my C code so that instead of being a Python
> module, it runs as a standalone binary, and it works as it should.
> Calling it with os.spawn* works. The two versions are essentially the
> same, the primary differences being the necessary difference in how
> arguments and return values are handled.
>
> This will work if necessary, but I would think having it as a Python
> module would be slightly more elegant and efficient since we avoid the
> overhead of setting up new processes.[/color]

Mybe you can build your C code as shared library and use `ctypes` to call
the functions from Python:

http://starship.python.net/crew/theller/ctypes/

Ciao,
Marc 'BlackJack' Rintsch

[david@quanterium.zzn.com]

Re: Are there memory limits for external C modules?

david@quanteriu m.zzn.com wrote:[color=blue]
> I was able to modify my C code so that instead of being a Python
> module, it runs as a standalone binary, and it works as it should.
> Calling it with os.spawn* works. The two versions are essentially the
> same, the primary differences being the necessary difference in how
> arguments and return values are handled.
>
> This will work if necessary, but I would think having it as a Python
> module would be slightly more elegant and efficient since we avoid the
> overhead of setting up new processes.[/color]

I was able to resolve this issue. I was missing a call into Ethereal
that allocated necessary memory. Now the Python module version works,
and some various GLib assert warnings I was getting have gone away as
well.

Some testing has shown that the Python module version is quicker; I'm
assuming that the overhead involved in setting up a separate process
was slowing things down. There was a noticeable difference.

- David