managing multiple subprocesses

[Marcos]

Hi guys,
I realise this question has been answered in one form or another many
times before but I can't quite find the solution I need. I am trying to
run multiple subprocesses from a python script and then wait until all
subprocesses have completed before continuing. The subprocesses run on
other machines ie this is a coarse grained parallel computation task.

So the commands I am using are:
for x in range(0, limit):
os.system("xgri d .... ./someprogram %d &" % x)

and then to check whether they are all finished I do a very crude:

joblist = commands.getout put("xgrid .... -job list")
while joblist != "{jobArray = (); }":
time.sleep(1)
joblist = commands.getout put("xgrid .... -job list")

Xgrid is a system for parallel computation on Mac OS X. It is quite
good, check it out at www.apple.com/acg/xgrid. But anyway the method I
am using above, while it works, it quite often hangs. I'd like to just
run all the subprocesses at once and then check within the script,
rather than an auxilliary, to see if everything has returned. The
output of the commands is not important as that gets redirected to
files outside the script. I have tried all sorts of popens / excevs /
os.systems / commands etc etc. I realise the subprocess module may have
what I need but I can't get python 2.4 on the Mac so I need a 2.3 based
solution. Any help is much appreciated. Cheers.

[Oren Tirosh]

Re: managing multiple subprocesses

"Marcos" <seejadee@gmail .com> wrote in message news:<110738941 6.531437.227160 @l41g2000cwc.go oglegroups.com> ...[color=blue]
> ...
> os.systems / commands etc etc. I realise the subprocess module may have
> what I need but I can't get python 2.4 on the Mac so I need a 2.3 based
> solution. Any help is much appreciated. Cheers.[/color]

The Python 2.4 subprocess module by Peter Astrand is pure python for
posix systems (a small C extension module is required only for win32
systems). You can bundle it with your code and use it with older
Python versions. It even contains backward compatibility code to
replace True and False if not found so it can run on Python 2.2.

Oren

[Fredrik Lundh]

Re: managing multiple subprocesses

Oren Tirosh wrote:
[color=blue][color=green]
>> os.systems / commands etc etc. I realise the subprocess module may have
>> what I need but I can't get python 2.4 on the Mac so I need a 2.3 based
>> solution. Any help is much appreciated. Cheers.[/color]
>
> The Python 2.4 subprocess module by Peter Astrand is pure python for
> posix systems (a small C extension module is required only for win32
> systems). You can bundle it with your code and use it with older
> Python versions. It even contains backward compatibility code to
> replace True and False if not found so it can run on Python 2.2.[/color]

footnote: a stand-alone distribution is available here:

http://www.lysator.liu.se/~astrand/popen5/

direct link to the latest subprocess.py source:

http://tinyurl.com/3mmka

</F>