build flow? SCons? AAP? process creation?

[corey.coughlin@comcast.net]

Hey guys, here's a little question. I'm looking for something like a
build system, where I can set up a bunch of jobs with dependencies, and
have them run in parallel. Ideally, I'd like a system where jobs can
be run in parallel, all the stdout and stderr for each job is kept in a
database of some kind, jobs can be anything that can be run at a
command line, and a failing job won't cause the whole run to fail.
I've been looking into some of the standard stuff (GNU make) and some
of the more pythonic stuff like SCons and AAP. SCons seems to do
parallel execution, which is nice, but I can't figure out from the docs
if it's possible to keep stdout and stderr somewhere. AAP seems to
have a nice interface, but it doesn't do parallel.

So if I do wind up having to write this thing myself, I've been
checking the docs on process creation, and have a couple questions if
anyone can fill me in. It looks like the os.spawn* commands can start
nonblocking sub-processes, but there doesn't seem to be a way to get
stdout and stderr. On the other hand, the popen commands make it easy
to trap stdout and stderr, but I guess I'd have to do the thread setup
and spawning myself. Is there another alternative that I'm missing
here?

Thanks!

[Dennis Benzinger]

Re: build flow? SCons? AAP? process creation?

corey.coughlin@ comcast.net schrieb:[color=blue]
> [...]
> So if I do wind up having to write this thing myself, I've been
> checking the docs on process creation, and have a couple questions if
> anyone can fill me in. It looks like the os.spawn* commands can start
> nonblocking sub-processes, but there doesn't seem to be a way to get
> stdout and stderr. On the other hand, the popen commands make it easy
> to trap stdout and stderr, but I guess I'd have to do the thread setup
> and spawning myself. Is there another alternative that I'm missing
> here?
> [...][/color]

Yes, you are missing the subprocess module which was introduced in
Python 2.4: http://docs.python.org/lib/module-subprocess.html

Bye,
Dennis

[corey.coughlin@comcast.net]

Re: build flow? SCons? AAP? process creation?

good point, I hadn't checked the docs too closely. Shame it's only in
2.4 though, we're still running 2.2 around here, but it would be nice
to have a reason to upgrade, anyway. Thanks for the pointer!

[Steve Christensen]

Re: build flow? SCons? AAP? process creation?

In article <1113420994.118 316.245190@g14g 2000cwa.googleg roups.com>, corey.coughlin@ comcast.net wrote:[color=blue]
> Hey guys, here's a little question. I'm looking for something like a
> build system, where I can set up a bunch of jobs with dependencies, and
> have them run in parallel. Ideally, I'd like a system where jobs can
> be run in parallel, all the stdout and stderr for each job is kept in a
> database of some kind, jobs can be anything that can be run at a
> command line, and a failing job won't cause the whole run to fail.
> I've been looking into some of the standard stuff (GNU make) and some
> of the more pythonic stuff like SCons and AAP. SCons seems to do
> parallel execution, which is nice, but I can't figure out from the docs
> if it's possible to keep stdout and stderr somewhere. AAP seems to
> have a nice interface, but it doesn't do parallel.[/color]

Here's something from the SCons Wiki:

http://www.scons.org/cgi-bin/wiki/BuildLog

-Steve

--
So if you people can't figure out what my birthday is based on subtle
hints and vague statements made at random intervals then I'm beginning
to think that you really don't care. -- Dean Lenort, a.r.k.