Best way to spawn process on back end computer

[sophie_newbie]

Hi,

I'm running a python cgi script on a frontend web server and I want it
to spawn another script (that takes a long time to run) on a backend
number crunching server thats connected to the same network. What do
you think is the best way to do this? I have a few ideas but I'm sure
there is a "best" way to go about this.

Thanks.

[Simon Brunning]

Re: Best way to spawn process on back end computer

2008/10/16 sophie_newbie <paulgeeleher@g mail.com>:

I'm running a python cgi script on a frontend web server and I want it
to spawn another script (that takes a long time to run) on a backend
number crunching server thats connected to the same network. What do
you think is the best way to do this? I have a few ideas but I'm sure
there is a "best" way to go about this.

http://edit.kamaelia.org/ might be qworth a look.

--
Cheers,
Simon B.
simon@brunningo nline.net

[Robin Becker]

Re: Best way to spawn process on back end computer

sophie_newbie wrote:

Hi,
>
I'm running a python cgi script on a frontend web server and I want it
to spawn another script (that takes a long time to run) on a backend
number crunching server thats connected to the same network. What do
you think is the best way to do this? I have a few ideas but I'm sure
there is a "best" way to go about this.

The main problem here is that you'll probably need to detach the job to allow
the current cgi request to return a response to the client.

The implication of that is that the job either has to be anonymous and requires
no further attention or you need to provide some means of making the job
responsive to requests about its status so that a periodic request can be made
by the web page. That implies that the job can be identified and the creation
reponse returns the identity. One of the major problems is that the normal www
user has few privileges and cannot normally write to disk.

I have done this using both external shell scripts to do the main processing and
detaching and or python scripts that know how to detach. It was not terribly
easy or obvious.

Another alternative, as Simon's Kamaelia might indicate, is that you might
consider running a job server to service the cgi script requests on the remote
host. I have also done this as part of a web application. One of the advantages
was that the jobserver can run as any user and thus gets access to whatever the
owner has; additionally by providing a suitable protocol eg XMLRPC you can test
the jobserver without going through the web.
--
Robin Becker

[Michael Sparks]

Re: Best way to spawn process on back end computer

sophie_newbie wrote:

Hi,
>
I'm running a python cgi script on a frontend web server and I want it
to spawn another script (that takes a long time to run) on a backend
number crunching server thats connected to the same network. What do
you think is the best way to do this? I have a few ideas but I'm sure
there is a "best" way to go about this.

Best is always subjective - what sort of interaction are you after?

Web request to trigger an activity (fire and forget)

Web request to trigger an activity, which produces output suitable for
something else?

Web request to trigger an activity, which produces output after a while that
you want to send back over a later web request?

Web request to trigger an activity, which produces output after a while that
you want to send back over a later web request, and you want to easily tie
the two things together?

Web request to trigger an activity, which produces output after a while that
you want to send back over a later web request, and you want those two
events to look like part of one operation?

Picking a concrete example...

Or something a bit more like:
* User uploads an image
* Gets converted to a bunch of standard sizes, and placed into a queue
for moderation and then later use?

Or perhaps:
* User uploads a video
* That gets transcoded to a different video format ? (eg video
contribution transcoded to flash video?)

In the latter two cases, this is something I've needed to do, so I created a
simple WSGI filter that dumps the uploaded images/videos into a standard
file location, and then had a script that watches that standard location
for new images & videos, and does those conversions and transcodes.

Strictly speaking to be "clean" about it, you need to dump the file to a
temporary location whilst you're grabbing it from the network and when it's
been written to disk rename it.

I've no idea if this maps to your problem at all. (since I can think of a
few scenarios that match your description).

If it does match, then this code may be handy:

http://code.google.com/p/kamaelia/source/browse/branches/private_MPS_Participate/Apps/FileProcessor/App/BatchFileProcessor.py

If you want to use that code, let me know and I'll package it up. There's an
older version here:
* http://www.kamaelia.org/release/Kama...r-0.1.0.tar.gz

Regards,


Michael.
--

http://www.kamaelia.org/Home