Recently a post that mentioned a recipe that extended subprocess to
allow killable processes caused me to do some thinking. Some of my
larger bash scripts are starting to become a bit unwieldy (hundreds of
lines of code). Yet for many things bash just works out so well because
it is so close to the file system and processes. As part of another
project, I now have need of a really good library to make it almost as
easy to do things in Python as it is in Bash. With a simple wrapper
around subprocess, I'm pretty much able to do most things. Most of my
complicated bash hackery involves using awk, sed, grep, and cut to
process text, which python does quite nicely, thank you very much. But
there's a few things to add.
To wit, I'm wanting to write a library that can deal with the following
things:
- spawn a process, feed it std in, get stdout, stderr, and err code.
This is largely already accomplished by subprocess
- spawn off processes as background daemons
- spawn multiple processes and pipe output to input.
- can do fancier things like bash does, like combine stderr/stdout,
switch stderr/stdout, redirects to and from files
- transparently allow a python function or object to be a part of
the pipeline at any stage.
Questions include, how would one design the interface for things, like
assembling pipes? Several ideas include:
pipe([prog1,args],[prog2,args],...)
or
run([prog1,args]).pipe([prog2,args]).pipe(...)
The former doesn't deal very well with re-plumbing of the pipes, nor is
there an easy way to redirect to and from a file. The second syntax is
more flexible but a bit cumbersome. Also it doesn't allow redirection
or flexible plumbing either.
Any ideas on how I could design this?
allow killable processes caused me to do some thinking. Some of my
larger bash scripts are starting to become a bit unwieldy (hundreds of
lines of code). Yet for many things bash just works out so well because
it is so close to the file system and processes. As part of another
project, I now have need of a really good library to make it almost as
easy to do things in Python as it is in Bash. With a simple wrapper
around subprocess, I'm pretty much able to do most things. Most of my
complicated bash hackery involves using awk, sed, grep, and cut to
process text, which python does quite nicely, thank you very much. But
there's a few things to add.
To wit, I'm wanting to write a library that can deal with the following
things:
- spawn a process, feed it std in, get stdout, stderr, and err code.
This is largely already accomplished by subprocess
- spawn off processes as background daemons
- spawn multiple processes and pipe output to input.
- can do fancier things like bash does, like combine stderr/stdout,
switch stderr/stdout, redirects to and from files
- transparently allow a python function or object to be a part of
the pipeline at any stage.
Questions include, how would one design the interface for things, like
assembling pipes? Several ideas include:
pipe([prog1,args],[prog2,args],...)
or
run([prog1,args]).pipe([prog2,args]).pipe(...)
The former doesn't deal very well with re-plumbing of the pipes, nor is
there an easy way to redirect to and from a file. The second syntax is
more flexible but a bit cumbersome. Also it doesn't allow redirection
or flexible plumbing either.
Any ideas on how I could design this?
Comment