Non-blocking pipes during subprocess handling

[Tom Plunket]

I'm using subprocess to launch, well, sub-processes, but now I'm
stumbling due to blocking I/O.

Is there a way for me to know that there's data on a pipe, and possibly
how much data is there so I can get it? Currently I'm doing this:

process = subprocess.Pope n(
args,
bufsize=1,
universal_newli nes=True,
stdout=subproce ss.PIPE,
stderr=subproce ss.PIPE)

def ProcessOutput(i nstream, outstream):
text = instream.readli ne()
if len(text) 0:
print >>outstream, text,
return True
else:
return False

while process.poll() is None:
ProcessOutput(p rocess.stdout, sys.stdout)
ProcessOutput(p rocess.stderr, sys.stderr)

# clean up everything to EOF once the process ends.
somethingPrinte d = True
while somethingPrinte d:
somethingPrinte d = ProcessOutput(
process.stdout, sys.stdout)
somethingPrinte d |= ProcessOutput(
process.stderr, sys.stderr)


Unfortunately, stream.readline will block 'til it gets a line, and
typically there won't be anything on the stderr stream. The reason for
the redirections in the first place is that I'm launching this script as
a subprocess from a GUI app that catches stdout and stderr and directs
the output to the appropriate windows, but in some cases I don't
actually want the output at all (I've removed that logic though since it
needlessly complicates my example; suffice to say everything below the
process = subprocess.Pope n... line is enclosed in a try and then in an
if block.

The documentation on file.read() indicate that there's an option for
"non-blocking" mode, but I'm stumped as to how to even look for how to
enable and use that.

thanks,
-tom!

--

[Gabriel Genellina]

Re: Non-blocking pipes during subprocess handling

At Monday 8/1/2007 22:09, Tom Plunket wrote:

>I'm using subprocess to launch, well, sub-processes, but now I'm
>stumbling due to blocking I/O.
>
>Is there a way for me to know that there's data on a pipe, and possibly
>how much data is there so I can get it? Currently I'm doing this:

Using a thread for each stream is the safest way, specially if you
can't control the child process.


--
Gabriel Genellina
Softlab SRL






_______________ _______________ _______________ _____
Preguntá. Respondé. Descubrí.
Todo lo que querías saber, y lo que ni imaginabas,
está en Yahoo! Respuestas (Beta).
¡Probalo ya!

http://www.yahoo.com.ar/respuestas

[Nick Craig-Wood]

Re: Non-blocking pipes during subprocess handling

Tom Plunket <tomas@fancy.or gwrote:

I'm using subprocess to launch, well, sub-processes, but now I'm
stumbling due to blocking I/O.
>
Is there a way for me to know that there's data on a pipe, and possibly
how much data is there so I can get it?

You might want to check out this modification to subprocess which does
non-blocking pipes.

http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554

I personally think something like that should be built into subprocess

--
Nick Craig-Wood <nick@craig-wood.com-- http://www.craig-wood.com/nick

[Donn Cave]

Re: Non-blocking pipes during subprocess handling

In article <22q5q25jn7dfti 18rv1tshj7os5a0 gm27r@4ax.com>,
Tom Plunket <tomas@fancy.or gwrote:

I'm using subprocess to launch, well, sub-processes, but now I'm
stumbling due to blocking I/O.
>
Is there a way for me to know that there's data on a pipe, and possibly
how much data is there so I can get it? Currently I'm doing this:
>
process = subprocess.Pope n(
args,
bufsize=1,
universal_newli nes=True,
stdout=subproce ss.PIPE,
stderr=subproce ss.PIPE)
>
def ProcessOutput(i nstream, outstream):
text = instream.readli ne()
if len(text) 0:
print >>outstream, text,
return True
else:
return False

I think it would be fair to say that your problem is
not due to blocking I/O, so much as buffered I/O. Since
you don't appear to need to read one line at a time, you
can detect and read data from the file descriptor without
any buffering. Don't mix with buffered I/O, as this will
throw the select off. From memory - better check, since
it has been a while since I wrote anything real like this
(or for that matter much of anything in Python) --


import select
def ProcessOutput(i nstream, outstream):
fdr = [instream.fileno ()]
(r, w, e) = select.select(f dr, [], [], 0.0)
for fd in r:
text = os.read(fd, 4096)
outstream.write (text)

Donn Cave, donn@u.washingt on.edu