calling Pyrex results from C

[Kyler Laird]

I need to submit C/C++ code for a class. (It's not a programming
class. The choice of language is inertial. I think that it mostly
serves to distract students from the course subject.) I'm fairly
fluent with C but it hurts to think about writing in C when Python
is *so* much more appropriate for these operations.

I'd like to keep my sanity and satisfy the course requirements by
programming in Python and converting the code to C. It looks like
a few people have scratched this itch already, but of the
translators I found, most of them (Python2C, P2C, PyFront) seem to
be dead. Pyrex, however, appears to be well-maintained and is even
available as a Debian package.

http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/

I realize that the goal of Pyrex is to create a module that can be
called from Python. For my need of calling the result from C, the
other utilities are probably more appropriate but I think that
Pyrex could be useful to me for less contrived situations than this
so it's worthwhile to learn more about it.

http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/version/Doc/overview.html#InterfacingWithExternal

You can also use public declarations to make C functions
and variables defined in a Pyrex module available to
external C code. The need for this is expected to be less
frequent, but you might want to do it, for example, if you
are embedding Python in another application as a scripting
language. Just as a Pyrex module can be used as a bridge to
allow Python code to call C code, it can also be used to
allow C code to call Python code.

[...]

You can make C variables and functions defined in a Pyrex
module accessible to external C code (or another Pyrex
module) using the public keyword

I've discovered that as long as everything in the code is cdef-ed,
I can call Pyrex code from my C code. If, however, I so much as
mention anything Pythonish, it segfaults during execution.

For example, this is fine.
cdef public int foo():
cdef int i
i = 123

return(i)

And this results in a segfault.
cdef public int foo():
cdef int i
i = 123
j = 5

return(i)

This means that I can't, for example, make a function that takes a
filename (string), uses PIL to do a bunch of operations, and
returns a string.

Any suggestions (besides "suck it up and do it all in C")?

Thank you.

--kyler

[François Pinard]

Re: calling Pyrex results from C

[Kyler Laird]
[color=blue]
> I realize that the goal of Pyrex is to create a module that can be
> called from Python.[/color]

Or vice-versa. I found Pyrex useful for calling Python modules from
within C programs which were not Python-aware to start with. You may
even write wholly Pyrex programs.
[color=blue]
> I've discovered that as long as everything in the code is cdef-ed,
> I can call Pyrex code from my C code. If, however, I so much as
> mention anything Pythonish, it segfaults during execution.[/color]

There is some magic needed to establish Python context, not much.
Hoping that it will help, here is a template I use when generating
Pyrex main programs. `%(module)s' shall be replaced by the Pyrex module
name, as Pyrex needs to initialise itself. When Python calls Pyrex,
initialisation occurs automatically at `import' time, but when C
initially calls Pyrex, initialisation should be explicitly launched.



cdef extern void Py_Initialize()
cdef extern void Py_Finalize()
cdef extern void init%(module)s( )

cdef extern int main(int argc, char **argv):
cdef int index, status
Py_Initialize()
init%(module)s( )
import sys
try:
arguments = []
for index from 0 <= index < argc:
arguments.appen d(argv[index])
sys.argv = arguments
result = main_applicatio n()
raise SystemExit, result or 0
except SystemExit, result:
result = str(result)
try:
status = int(result)
except ValueError:
if not result.endswith ('\n'):
result = result + '\n'
sys.stderr.writ e(result)
status = 1
except:
import traceback
traceback.print _exc()
status = 1
Py_Finalize()
return status


--
François Pinard http://www.iro.umontreal.ca/~pinard

[John J. Lee]

Re: calling Pyrex results from C

Kyler Laird <Kyler@news.Lai rds.org> writes:
[...][color=blue]
> This means that I can't, for example, make a function that takes a
> filename (string), uses PIL to do a bunch of operations, and
> returns a string.[/color]

I don't see why you shouldn't be able to write a module in Pyrex, then
use standard embedding techniques to embed Python and then call your
module with a few C API calls. Or do the same, but also write a Pyrex
cdef function to wrap it all up, so you can just call that as a "plain
old" C function instead of those final few C API calls (you'll still
need other embedding calls, of course, to initialise Python, etc.).

Try the pyrex mailing list if you're stuck.

I'd very definitely check to make sure you aren't going to get marked
down for being a smartass, though (even if you're generating C that
doesn't depend on Python at all, it might still be unreadable).

[color=blue]
> Any suggestions (besides "suck it up and do it all in C")?[/color]

I think I'd do it in C, regardless of whether or not it's allowable to
use Pyrex.


John

[Kyler Laird]

Re: calling Pyrex results from C

On Tue, Jan 20, 2004 at 12:27:33PM -0500, Fran?ois Pinard wrote:
[color=blue]
> There is some magic needed to establish Python context, not much.
> Hoping that it will help, here is a template I use when generating
> Pyrex main programs.[/color]

Paul started me on this path and you've given me exactly what I need to
keep going.

Thank you!

--kyler

[Kyler Laird]

Re: calling Pyrex results from C

Paul Prescod graciously responded to my plea for help by gently asking
if I had initialized Python. Doh! I had gotten lazy and assumed that
Pyrex would take care of such details. (How could it?!)

Initializing the interpreter seems to have the simple test case I made
working smoothly. Now it's on to bigger and better things...

Pyrex is *so* cool.

Thank you, Paul!

--kyler

[François Pinard]

Re: calling Pyrex results from C

[Kyler Laird][color=blue]
> On Tue, Jan 20, 2004 at 12:27:33PM -0500, Fran?ois Pinard wrote:[/color]
[color=blue][color=green]
> > There is some magic needed to establish Python context, not much.
> > Hoping that it will help, here is a template I use when generating
> > Pyrex main programs.[/color][/color]
[color=blue]
> Paul started me on this path and you've given me exactly what I need to
> keep going.[/color]

Good. :-)

There is also some linking magic, which differs from system to system,
it might be documented in the Python Embedding API, I'm not sure. If
you do not stumble on the proper recipe, I'll share what I use on Linux.
Just ask if you want me to go search into some of my Makefiles :-).

--
François Pinard http://www.iro.umontreal.ca/~pinard

[Kyler Laird]

Re: calling Pyrex results from C

The help I've gotten in this thread has enabled me to complete my
assignment...we ll, at least I can generate C code that spits out
the answer.

I'd like to factor out more of the Python so that my handwritten
C code can do more of the high-level operations. I'm having some
difficulties passing data between my C and the Pyrex code.

One of the first things I thought I'd build is a helper function
to return a string representation of an object. I thought that
this would be fairly straightforward . Unfortunately it segfaults
when called with some Python objects.

Pyrex:
cdef public image_file_open (char* image_filename) :
return(Image.op en(image_filena me))

cdef public image_size(imag e_PIL):
return(Image.si ze)

cdef public char* string(x):
s = str(x)
return(s)

my C:
void *im, *im_size;
im = image_file_open (input_filename );
im_size = image_size(im);
printf("im=%s\n ", string(im));
printf("im_size =%s\n", string(im_size) );

The first call to string() succeeds but the second one fails. I
suspect that I've expected too much of Pyrex again. Do I need to
allocate memory, manipulate reference counts or something like
that?

It's difficult for me to get back to thinking in terms of
passing around data in C ways. I almost forgot what a boon
tuples have been. (Perhaps I need to code in assembly for
awhile to get back in that frame of mind.) Someday I might try
to pass structures between C and Pyrex. I'm slowly getting
comfortable with Pyrex but I can see how my code can get *much*
cleaner once I understand more of its capabilities.

Thank you.

--kyler

[Paul Prescod]

Re: calling Pyrex results from C

Kyler Laird wrote:
[color=blue]
>
> Pyrex:
> cdef public image_file_open (char* image_filename) :
> return(Image.op en(image_filena me))
>
> cdef public image_size(imag e_PIL):
> return(Image.si ze)
>
> cdef public char* string(x):
> s = str(x)
> return(s)
>
> my C:
> void *im, *im_size;
> im = image_file_open (input_filename );
> im_size = image_size(im);
> printf("im=%s\n ", string(im));
> printf("im_size =%s\n", string(im_size) );
>
> The first call to string() succeeds but the second one fails. I
> suspect that I've expected too much of Pyrex again. Do I need to
> allocate memory, manipulate reference counts or something like
> that?[/color]

First, I'd suggest that "integer" is a perfectly good type in both C and
Pyrex so you shouldn't pass around Python objects representing integers.

Second, you aren't checking the return codes of your functions and C has
no exception handling, tracebacks, etc. image_size is probably returning
0 because it is probably throwing an exception because you are asking
for the size attribute of the Image class rather than the image_PIL object.

You actually would have gotten a Pyrex error on your console if your
function were defined to return "int" or "void" because Pyrex would KNOW
that there's no way you are doing exception handling so it would try to
compensate by printing exceptions to the console. But I wouldn't depend
on that feature because it doesn't help for functions that really should
return objects. Better to check your return values.

Once I fix the Image/Image_PIL error your code runs okay on my computer.
But you are walking on thin ice and may just be lucky. It is simply
not possible to work with strings generated at runtime in C without
worrying about memory allocation sometime. In this case I have the
strong suspicion that the string() function is either creating and
destroying a string object and then returning you a pointer to the dead
object's internal memory buffer (bad news!) or simply losing a reference
to the (still living) string object: still not a good thing. Eyeballing
the code I believe the former is the issue. You could check for sure by
adding some printf's to the generated code to look at __pyx_v_s->ob_refcnt.

Why not let Python do the "print" rather than using "printf".

Paul Prescod

[Kyler Laird]

Re: calling Pyrex results from C

On Wed, Jan 21, 2004 at 08:23:53AM -0800, Paul Prescod wrote:
[color=blue]
> First, I'd suggest that "integer" is a perfectly good type in both C and
> Pyrex so you shouldn't pass around Python objects representing integers.[/color]

That would be my inclination too if I just had an integer to pass.
Are you suggesting that I should pass tuples of integers as pointers
to (an array of) integers?
[color=blue]
> Second, you aren't checking the return codes of your functions and C has
> no exception handling, tracebacks, etc. image_size is probably returning
> 0 because it is probably throwing an exception because you are asking
> for the size attribute of the Image class rather than the image_PIL object.[/color]

I'm trying to keep my examples simple so that it's easy to pick out the
mistakes.
[color=blue]
> Once I fix the Image/Image_PIL error your code runs okay on my computer.[/color]

Grrr...yup, that does it. Dumb mistake - the result of pulling code
out of various places and gluing it together without checks.
[color=blue]
> But you are walking on thin ice and may just be lucky. It is simply
> not possible to work with strings generated at runtime in C without
> worrying about memory allocation sometime. In this case I have the
> strong suspicion that the string() function is either creating and
> destroying a string object and then returning you a pointer to the dead
> object's internal memory buffer (bad news!) or simply losing a reference
> to the (still living) string object: still not a good thing. Eyeballing
> the code I believe the former is the issue. You could check for sure by
> adding some printf's to the generated code to look at __pyx_v_s->ob_refcnt.[/color]

I feared this was a problem. I'll play with it.
[color=blue]
> Why not let Python do the "print" rather than using "printf".[/color]

Printing is not always my final goal.

Thank you.

--kyler

[Kyler Laird]

Re: calling Pyrex results from C

Paul Prescod <paul@prescod.n et> writes:
[color=blue]
> 1. How is your teacher going to react when you hand in obfuscated-ish
> C code that depends on the whole Python interpreter _and_ PIL?[/color]

Shhh...don't tell anyone.
Grades for ECE661 as of Wed Feb 4 15:01:35 2004
Name hw1 hw2
---- ----- -----
Maximums: 100.0 100.0
Laird, Kyler 100.0 100.0
Class Average: 93.8 86.6
Class Standard Deviation: 20.1 24.3

This is Computer Vision, not Introduction to Pointers and Memory Allocation.
Pyrex frees me to concentrate on the course material.

(And I *need* to concentrate on the course material. I'm in a class full of
incredibly smart people with a lot more background in the subject. Pyrex
helps me keep up with them. So does only taking one class and investing
ridiculous amounts of time on the homework.)

Thanks again for helping me avoid C.

Now to write some "thinning" code...

--kyler

[Greg Ewing (using news.cis.dfn.de)]

Re: calling Pyrex results from C

Kyler Laird wrote:[color=blue]
> cdef public char* string(x):
> s = str(x)
> return(s)[/color]

Whenever you allow Pyrex to coerce a Python string into a
char *, you need to be careful that the Python string will
remain alive for as long as the C pointer is needed.

In this case, str(x) is probably returning a string with
no other references, so it's deallocated as soon as the
function finishes, and a dangling pointer is then returned.

You'll need to make a copy of the string's contents
somehow, e.g.

cdef extern from "string.h":
char *strcpy(char *)

cdef public char* string(x):
s = str(x)
return strcpy(s)

and remember to free() it when you're finished with
it.

(Warning: Don't try to write it as

return strcpy(str(x)) # WRONG

or you'll have the same problem -- the string will
be deallocated before strcpy() gets a chance to
copy it!)

Some other comments:

* In the C code, it's better to declare pointers to
Python objects as PyObject * rather than void *, e.g.

PyObject *im, *im_size;

since this will give you better type checking from the
C compiler.

* Your C code as written leaks Python objects, since it
never decrefs the objects returned by image_file_open
and image_size.It also doesn't do anything about checking
for and handling Python exceptions.

It's much better if you can do all creation and manipulation
of Python objects in Pyrex, since it takes care of
refcounting and error checking automatically (although
read the doc section on Error Return Values to make sure
exceptions are propagated from functions that don't
return Python objects).

You may want to put something in your main function to
catch exceptions and print a traceback, since that
won't happen automatically with no Python interpreter
at the top level.

--
Greg Ewing, Computer Science Dept,
University of Canterbury,
Christchurch, New Zealand

http://www.cosc.canterbury.ac.nz/~greg