pythonic exec* spawn*

[Daniel Nogradi]

Is it possible to pass a python object to a python program as argument?

In my program I would like to start executing an other python program
and don't wait until that finishes, only launch it and keep running
the original program. The obvious way I can think of it is using the
exec* or spawn* function families. However, these are generic
functions to start any external program and I was wondering if there
was a special way to start a python program (as opposed to an
arbitrary program), especially because I would like to pass data to
this other program in the form of python objects.

While the original program is running it creates all sorts of data in
a nice pythonic way, for example as a class instance with lot of
member data, and then I wouldn't want to convert all of this into
command line arguments because that would create a huge list. I would
like to pass the whole object at once to the second python program
which should start doing its thing with it while the original program
should keep running.

I was looking around for threading and forking but couldn't come up
with a clear solution.

[kmkz89@gmail.com]

Re: pythonic exec* spawn*

You could possibly pickle an object and send it, but that could be a
bit messy...

[Jeremiah]

Re: pythonic exec* spawn*

you can use the threads commands

[dwelch]

Re: pythonic exec* spawn*

kmkz89@gmail.co m wrote:[color=blue]
> You could possibly pickle an object and send it, but that could be a
> bit messy...
>[/color]
pickle it (or shelve?) and then pass the pickle/shleve filename to the
other process as a command line parameter?

Not sure if this would work or not, but that's where I would start.

-Don

[Rene Pijlman]

Re: pythonic exec* spawn*

Daniel Nogradi:[color=blue]
>I would like to pass the whole object at once to the second python
>program which should start doing its thing with it while the original
>program should keep running.[/color]

os.fork() does that (on Mac and Unix).

--
René Pijlman

[Daniel Nogradi]

Re: pythonic exec* spawn*

> >I would like to pass the whole object at once to the second python[color=blue][color=green]
> >program which should start doing its thing with it while the original
> >program should keep running.[/color]
>
> os.fork() does that (on Mac and Unix).
>[/color]

Okay, but how? It seems to me that if the process which issued
os.fork() ends, then the forked process also ends. And what I would
need is that regardless of the execution time of the original program,
the second one which the original starts (or forks) should finish what
it supposed to do. Roughly I have the following in mind:

-------------------------------------------------------------------------------
def longer( data ):
# do some stuff, that takes long


# do something to collect data

data = something

# here call the function longer( data ) in a way
# that execution of the code doesn't stop, so
# we get to the following print and then to the end

print "okay, the first program finished"
------------------------------------------------------------------------------

But the execute of longer( data ) should keep going even though the
original program ended. I'm pretty sure it's something basic and
probably I'm not aware of the right concepts and that's why I can't
find the right place to look in the docs.

[Rene Pijlman]

Re: pythonic exec* spawn*

Daniel Nogradi:[color=blue][color=green]
>> os.fork() does that (on Mac and Unix).[/color]
>
>Okay, but how?[/color]

Sorry, fork() is implemented strictly on a 'need to know' basis :-)
[color=blue]
>It seems to me that if the process which issued os.fork() ends, then
>the forked process also ends.[/color]

No, no, they're not a quantum mechanic photon pair. Every process decides
for itself if and when to end. As long as it's not signalled/killed, that
is.
[color=blue]
>But the execute of longer( data ) should keep going even though the
>original program ended. I'm pretty sure it's something basic and
>probably I'm not aware of the right concepts and that's why I can't
>find the right place to look in the docs.[/color]

You can search for "daemonize" .

Here's a recipe:

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/66012

The concept is best explained in this book:

http://www.kohala.com/start/apue.html

--
René Pijlman

[Daniel Nogradi]

Re: pythonic exec* spawn*

> >> os.fork() does that (on Mac and Unix).[color=blue][color=green]
> >
> >Okay, but how?[/color]
>
> Sorry, fork() is implemented strictly on a 'need to know' basis :-)
>[color=green]
> >It seems to me that if the process which issued os.fork() ends, then
> >the forked process also ends.[/color]
>
> No, no, they're not a quantum mechanic photon pair. Every process decides
> for itself if and when to end. As long as it's not signalled/killed, that
> is.
>[color=green]
> >But the execute of longer( data ) should keep going even though the
> >original program ended. I'm pretty sure it's something basic and
> >probably I'm not aware of the right concepts and that's why I can't
> >find the right place to look in the docs.[/color]
>
> You can search for "daemonize" .
>
> Here's a recipe:
> http://aspn.activestate.com/ASPN/Coo...n/Recipe/66012
>
> The concept is best explained in this book:
> http://www.kohala.com/start/apue.html[/color]

Okay, thanks a lot, I'll look into that.

[Daniel Nogradi]

Re: pythonic exec* spawn*

> > >> os.fork() does that (on Mac and Unix).[color=blue][color=green][color=darkred]
> > >
> > >Okay, but how?[/color]
> >
> > Sorry, fork() is implemented strictly on a 'need to know' basis :-)
> >[color=darkred]
> > >It seems to me that if the process which issued os.fork() ends, then
> > >the forked process also ends.[/color]
> >
> > No, no, they're not a quantum mechanic photon pair. Every process decides
> > for itself if and when to end. As long as it's not signalled/killed, that
> > is.
> >[color=darkred]
> > >But the execute of longer( data ) should keep going even though the
> > >original program ended. I'm pretty sure it's something basic and
> > >probably I'm not aware of the right concepts and that's why I can't
> > >find the right place to look in the docs.[/color]
> >
> > You can search for "daemonize" .
> >
> > Here's a recipe:
> > http://aspn.activestate.com/ASPN/Coo...n/Recipe/66012
> >
> > The concept is best explained in this book:
> > http://www.kohala.com/start/apue.html[/color][/color]

Thanks again, the recipe was very useful, and in the meantime I also
found this article,

http://www.informit.com/content/images/art_deitel_creatingprocesses/elementLinks/pythonhtp1_fork_article.pdf

which explains os.fork and os.wait in a python context. Maybe some
others will also make use of it, it clearified the basics for me.

[Nick Craig-Wood]

Re: pythonic exec* spawn*

Daniel Nogradi <nogradi@gmail. com> wrote:[color=blue]
> Is it possible to pass a python object to a python program as
> argument?[/color]

Yes - you can pickle it.

http://www.python.org/doc/2.4.2/lib/module-cPickle.html

You can pickle just about anything into a string and unpickle it back
into python objects. You shouldn't accept pickles from untrusted
sources as this may reperesent a security problem.
[color=blue]
> In my program I would like to start executing an other python program
> and don't wait until that finishes, only launch it and keep running
> the original program. The obvious way I can think of it is using the
> exec* or spawn* function families. However, these are generic
> functions to start any external program and I was wondering if there
> was a special way to start a python program (as opposed to an
> arbitrary program), especially because I would like to pass data to
> this other program in the form of python objects.
>
> While the original program is running it creates all sorts of data in
> a nice pythonic way, for example as a class instance with lot of
> member data, and then I wouldn't want to convert all of this into
> command line arguments because that would create a huge list. I would
> like to pass the whole object at once to the second python program
> which should start doing its thing with it while the original program
> should keep running.
>
> I was looking around for threading and forking but couldn't come up
> with a clear solution.[/color]

If you want to fork(), then I would use the subprocess module to
create a child process with a pipe to it. I would then pass python
objects as pickles back and forth across that pipe.

http://www.python.org/doc/2.4.2/lib/module-subprocess.html

You'll probably find threading easier though.

http://www.python.org/doc/2.4.2/lib/module-threading.html

But it will use your multiple CPUs less efficiently than fork()-ing.

--
Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick