calling python scripts as a sub-process

[Catherine Moroney]

I have one script (Match1) that calls a Fortran executable as a
sub-process, and I want to write another script (Match4) that
spawns off several instances of Match1 in parallel and then waits
until they all finish running. The only way I can think of doing this
is to call it as a sub-process, rather than directly.

I'm able to get Match1 working correctly in isolation, using the
subprocess.Pope n command, but calling an instance of Match1 as a
subprocess spawned from Match4 isn't working.

The command (stored as an array of strings) that I'm executing is:

['python ../src_python/Match1.py ',
'--file_ref=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_BF_F03_0024. hdf ',
'--file_cmp=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_DF_F03_0024. hdf ',
'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2'
", "--use_textid='tru e '"]

and I'm calling it as:

sub1 = subprocess.Pope n(command)

I get the error below. Does anybody know what this error refers
to and what I'm doing wrong? Is it even allowable to call another
script as a sub-process rather than calling it directly?

File "../src_python/Match4.py", line 24, in RunMatch4
sub1 = subprocess.Pope n(command1)
File "/usr/lib64/python2.5/subprocess.py", line 593, in __init__
errread, errwrite)
File "/usr/lib64/python2.5/subprocess.py", line 1051, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

Thanks for any help,

Catherine

[Philip Semanchuk]

Re: calling python scripts as a sub-process


On Nov 19, 2008, at 2:03 PM, Catherine Moroney wrote:

The command (stored as an array of strings) that I'm executing is:
>
['python ../src_python/Match1.py ', '--
file_ref=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_BF_F03_0024. hdf ',
'--file_cmp=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_DF_F03_0024. hdf
', '--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--
chmetric='M2' ", "--use_textid='tru e '"]
>

[snip]

>
I get the error below. Does anybody know what this error refers
to and what I'm doing wrong? Is it even allowable to call another
script as a sub-process rather than calling it directly?
>
File "../src_python/Match4.py", line 24, in RunMatch4
sub1 = subprocess.Pope n(command1)
File "/usr/lib64/python2.5/subprocess.py", line 593, in __init__
errread, errwrite)
File "/usr/lib64/python2.5/subprocess.py", line 1051, in
_execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

Try supplying a fully-qualified path to your script, e.g.:
['python /home/catherine/src_python/Match1.py ', '--
file_ref=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_BF_F03_0024. hdf ', '--
file_cmp=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_DF_F03_0024. hdf ', '--
block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2'
", "--use_textid='tru e '"]

[Mike Driscoll]

Re: calling python scripts as a sub-process

On Nov 19, 1:03 pm, Catherine Moroney
<Catherine.M.Mo ro...@jpl.nasa. govwrote:

I have one script (Match1) that calls a Fortran executable as a
sub-process, and I want to write another script (Match4) that
spawns off several instances of Match1 in parallel and then waits
until they all finish running.  The only way I can think of doing this
is to call it as a sub-process, rather than directly.
>
I'm able to get Match1 working correctly in isolation, using the
subprocess.Pope n command, but calling an instance of Match1 as a
subprocess spawned from Match4 isn't working.
>
The command (stored as an array of strings) that I'm executing is:
>
['python ../src_python/Match1.py ',
'--file_ref=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_BF_F03_0024. hdf ',
'--file_cmp=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_DF_F03_0024. hdf ',
'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2'
", "--use_textid='tru e '"]
>
and I'm calling it as:
>
sub1 = subprocess.Pope n(command)
>
I get the error below.  Does anybody know what this error refers
to and what I'm doing wrong?  Is it even allowable to call another
script as a sub-process rather than calling it directly?
>
  File "../src_python/Match4.py", line 24, in RunMatch4
     sub1 = subprocess.Pope n(command1)
   File "/usr/lib64/python2.5/subprocess.py", line 593, in __init__
     errread, errwrite)
   File "/usr/lib64/python2.5/subprocess.py", line 1051, in _execute_child
     raise child_exception
OSError: [Errno 2] No such file or directory
>
Thanks for any help,
>
Catherine

Try giving an absolute path to the python file rather than a relative
path. I don't think the subprocess module "knows" where to look
otherwise.

Mike

[Dan Upton]

Re: calling python scripts as a sub-process

On Wed, Nov 19, 2008 at 2:13 PM, Philip Semanchuk <philip@semanch uk.comwrote:

>
On Nov 19, 2008, at 2:03 PM, Catherine Moroney wrote:
>

>The command (stored as an array of strings) that I'm executing is:
>>
>['python ../src_python/Match1.py ',
>'--file_ref=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_BF_F03_0024. hdf ',
>'--file_cmp=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_DF_F03_0024. hdf ',
>'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2' ",
>"--use_textid='tru e '"]
>>

>
[snip]
>

>>
>I get the error below. Does anybody know what this error refers
>to and what I'm doing wrong? Is it even allowable to call another
>script as a sub-process rather than calling it directly?
>>
>File "../src_python/Match4.py", line 24, in RunMatch4
> sub1 = subprocess.Pope n(command1)
> File "/usr/lib64/python2.5/subprocess.py", line 593, in __init__
> errread, errwrite)
> File "/usr/lib64/python2.5/subprocess.py", line 1051, in _execute_child
> raise child_exception
>OSError: [Errno 2] No such file or directory

>
Try supplying a fully-qualified path to your script, e.g.:
['python /home/catherine/src_python/Match1.py ',
'--file_ref=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_BF_F03_0024. hdf ',
'--file_cmp=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_DF_F03_0024. hdf ',
'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2' ",
"--use_textid='tru e '"]

I think when I came across this error, I added shell=True, e.g.

sub1 = subprocess.Pope n(command, shell=True)

[Catherine Moroney]

Re: calling python scripts as a sub-process

I just tried that, and I get the same error.

Interestingly enough, a shorter (and incorrect) version
of the command works well enough so that it gets into the
Match1 code and does the argument check there.

The following code gets into Match1:

>>command = ['python', '../src_python/Match1.py','--filex="xyz"']
>>sub1 = subprocess.Pope n(command)

whereas this doesn't even get to call Match1:

command =
['python','/data/svn_workspace/cmm/sieglind/USC/EE569/tpaper/test/../src_python/Match1.py
', '--file_ref=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_BF_F03_0024. hdf ',
'--file_cmp=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_DF_F03_0024. hdf ',
'--block_start=62 ', '--block_end=62 ', '--istep=16 ', '--chmetric=M2 ',
'--use_textid=true ']

sub1 = subprocess.Pope n(command)

Can anybody see a reason for why the abbreviated version works, and
the full-up one doesn't?

Catherine

Philip Semanchuk wrote:

>
On Nov 19, 2008, at 2:03 PM, Catherine Moroney wrote:
>

>The command (stored as an array of strings) that I'm executing is:
>>
>['python ../src_python/Match1.py ',
>'--file_ref=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_BF_F03_0024. hdf ',
>'--file_cmp=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_DF_F03_0024. hdf ',
>'--block_start=62 ', '--block_end=62 ', '--istep=16 ',
>"--chmetric='M2' ", "--use_textid='tru e '"]
>>

>
[snip]
>

>>
>I get the error below. Does anybody know what this error refers
>to and what I'm doing wrong? Is it even allowable to call another
>script as a sub-process rather than calling it directly?
>>
>File "../src_python/Match4.py", line 24, in RunMatch4
> sub1 = subprocess.Pope n(command1)
> File "/usr/lib64/python2.5/subprocess.py", line 593, in __init__
> errread, errwrite)
> File "/usr/lib64/python2.5/subprocess.py", line 1051, in _execute_child
> raise child_exception
>OSError: [Errno 2] No such file or directory

>
Try supplying a fully-qualified path to your script, e.g.:
['python /home/catherine/src_python/Match1.py ',
'--file_ref=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_BF_F03_0024. hdf ',
'--file_cmp=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_DF_F03_0024. hdf ',
'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2'
", "--use_textid='tru e '"]
>
>
>

[Catherine Moroney]

Re: calling python scripts as a sub-process

Dan Upton wrote:

On Wed, Nov 19, 2008 at 2:13 PM, Philip Semanchuk <philip@semanch uk.comwrote:

>On Nov 19, 2008, at 2:03 PM, Catherine Moroney wrote:
>>

>>The command (stored as an array of strings) that I'm executing is:
>>>
>>['python ../src_python/Match1.py ',
>>'--file_ref=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_BF_F03_0024. hdf ',
>>'--file_cmp=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_DF_F03_0024. hdf ',
>>'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2' ",
>>"--use_textid='tru e '"]
>>>

>[snip]
>>

>>I get the error below. Does anybody know what this error refers
>>to and what I'm doing wrong? Is it even allowable to call another
>>script as a sub-process rather than calling it directly?
>>>
>>File "../src_python/Match4.py", line 24, in RunMatch4
>> sub1 = subprocess.Pope n(command1)
>> File "/usr/lib64/python2.5/subprocess.py", line 593, in __init__
>> errread, errwrite)
>> File "/usr/lib64/python2.5/subprocess.py", line 1051, in _execute_child
>> raise child_exception
>>OSError: [Errno 2] No such file or directory

>Try supplying a fully-qualified path to your script, e.g.:
>['python /home/catherine/src_python/Match1.py ',
>'--file_ref=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_BF_F03_0024. hdf ',
>'--file_cmp=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_DF_F03_0024. hdf ',
>'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2' ",
>"--use_textid='tru e '"]

>
I think when I came across this error, I added shell=True, e.g.
>
sub1 = subprocess.Pope n(command, shell=True)

I added the shell=True and this time it got into Match1 (hurrah!),
but it then opened up an interactive python session, and didn't
complete until I manually typed 'exit' in the interactive session.

Match1 looks like:

if __name__ == "__main__":
<<< parse arguments >>>

RunMatch1(file_ ref, file_cmp, iblock_start, iblock_end, \
nlinep, nsmpp, mindispx, maxdispx, mindispl, \
maxdispl, istep, chmetric, use_textid)

exit()

where the routine RunMatch1 does all the actual processing.

How do I get Match1 to run and exit normally without opening up an
interactive session, when called as a subprocess from Match4?

Catherine

[Dan Upton]

Re: calling python scripts as a sub-process

On Wed, Nov 19, 2008 at 2:38 PM, Catherine Moroney
<Catherine.M.Mo roney@jpl.nasa. govwrote:

Dan Upton wrote:

>>
>On Wed, Nov 19, 2008 at 2:13 PM, Philip Semanchuk <philip@semanch uk.com>
>wrote:

>>>
>>On Nov 19, 2008, at 2:03 PM, Catherine Moroney wrote:
>>>
>>>The command (stored as an array of strings) that I'm executing is:
>>>>
>>>['python ../src_python/Match1.py ',
>>>'--file_ref=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_BF_F03_0024. hdf ',
>>>'--file_cmp=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_DF_F03_0024. hdf ',
>>>'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2'
>>>",
>>>"--use_textid='tru e '"]
>>>>
>>[snip]
>>>
>>>I get the error below. Does anybody know what this error refers
>>>to and what I'm doing wrong? Is it even allowable to call another
>>>script as a sub-process rather than calling it directly?
>>>>
>>>File "../src_python/Match4.py", line 24, in RunMatch4
>>> sub1 = subprocess.Pope n(command1)
>>> File "/usr/lib64/python2.5/subprocess.py", line 593, in __init__
>>> errread, errwrite)
>>> File "/usr/lib64/python2.5/subprocess.py", line 1051, in _execute_child
>>> raise child_exception
>>>OSError: [Errno 2] No such file or directory
>>>
>>Try supplying a fully-qualified path to your script, e.g.:
>>['python /home/catherine/src_python/Match1.py ',
>>'--file_ref=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_BF_F03_0024. hdf ',
>>'--file_cmp=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_DF_F03_0024. hdf ',
>>'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2'
>>",
>>"--use_textid='tru e '"]

>>
>I think when I came across this error, I added shell=True, e.g.
>>
>sub1 = subprocess.Pope n(command, shell=True)

>
I added the shell=True and this time it got into Match1 (hurrah!),
but it then opened up an interactive python session, and didn't
complete until I manually typed 'exit' in the interactive session.
>
Match1 looks like:
>
if __name__ == "__main__":
<<< parse arguments >>>
>
RunMatch1(file_ ref, file_cmp, iblock_start, iblock_end, \
nlinep, nsmpp, mindispx, maxdispx, mindispl, \
maxdispl, istep, chmetric, use_textid)
>
exit()
>
where the routine RunMatch1 does all the actual processing.
>
How do I get Match1 to run and exit normally without opening up an
interactive session, when called as a subprocess from Match4?
>

Alternately, rather than using a list of arguments, have you tried
just using a string? (Again, that's the way I do it and I haven't
been having any problems recently, although I'm running shell scripts
or binaries with arguments rather than trying to invoke python on a
script.)

command = "python ../src_python/Match1.py
--file_ref=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_BF_F03_0024. hdf
--file_cmp=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_DF_F03_0024. hdf
--block_start=62 --block_end=62 --istep=16 --chmetric='M2'
--use_textid=true "

proc = subprocess.Pope n(command, shell=True)

[Jeremy Sanders]

Re: calling python scripts as a sub-process

Catherine Moroney wrote:

I have one script (Match1) that calls a Fortran executable as a
sub-process, and I want to write another script (Match4) that
spawns off several instances of Match1 in parallel and then waits
until they all finish running. The only way I can think of doing this
is to call it as a sub-process, rather than directly.
>
I'm able to get Match1 working correctly in isolation, using the
subprocess.Pope n command, but calling an instance of Match1 as a
subprocess spawned from Match4 isn't working.
>
The command (stored as an array of strings) that I'm executing is:
>
['python ../src_python/Match1.py ',
'--file_ref=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_BF_F03_0024. hdf ',
'--file_cmp=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_DF_F03_0024. hdf ',
'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2'
", "--use_textid='tru e '"]
>

If you want to avoid going by the shell, and you *should* for security
reasons, you need to have each of your arguments separately in the list
without the shell quoting and extra spaces, i.e.

['python', '../src_python/Match1.py',
'--file_ref=xxxx.h df', '--file_cmp=yyyy.h df',
'--block_start=xx' , '--block_end=62', '--istep=16', '--chmetric=M2',
'--use_texid=true']

Jeremy

--
Jeremy Sanders

http://www.jeremysanders.net/

[Jeremy Sanders]

Re: calling python scripts as a sub-process

Dan Upton wrote:

I think when I came across this error, I added shell=True, e.g.
>
sub1 = subprocess.Pope n(command, shell=True)

That's really papering over the bug. You need to have the parameters
separately, including the name of the program, separately in the list. You
need to remove any shell quoting you may use on the unix/dos command line.

Jeremy

--
Jeremy Sanders

http://www.jeremysanders.net/

[Catherine Moroney]

Re: calling python scripts as a sub-process

Dan Upton wrote:

On Wed, Nov 19, 2008 at 2:38 PM, Catherine Moroney
<Catherine.M.Mo roney@jpl.nasa. govwrote:

>Dan Upton wrote:

>>On Wed, Nov 19, 2008 at 2:13 PM, Philip Semanchuk <philip@semanch uk.com>
>>wrote:
>>>On Nov 19, 2008, at 2:03 PM, Catherine Moroney wrote:
>>>>
>>>>The command (stored as an array of strings) that I'm executing is:
>>>>>
>>>>['python ../src_python/Match1.py ',
>>>>'--file_ref=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_BF_F03_0024. hdf ',
>>>>'--file_cmp=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_DF_F03_0024. hdf ',
>>>>'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2'
>>>>",
>>>>"--use_textid='tru e '"]
>>>>>
>>>[snip]
>>>>
>>>>I get the error below. Does anybody know what this error refers
>>>>to and what I'm doing wrong? Is it even allowable to call another
>>>>script as a sub-process rather than calling it directly?
>>>>>
>>>>File "../src_python/Match4.py", line 24, in RunMatch4
>>>> sub1 = subprocess.Pope n(command1)
>>>> File "/usr/lib64/python2.5/subprocess.py", line 593, in __init__
>>>> errread, errwrite)
>>>> File "/usr/lib64/python2.5/subprocess.py", line 1051, in _execute_child
>>>> raise child_exception
>>>>OSError: [Errno 2] No such file or directory
>>>Try supplying a fully-qualified path to your script, e.g.:
>>>['python /home/catherine/src_python/Match1.py ',
>>>'--file_ref=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_BF_F03_0024. hdf ',
>>>'--file_cmp=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_DF_F03_0024. hdf ',
>>>'--block_start=62 ', '--block_end=62 ', '--istep=16 ', "--chmetric='M2'
>>>",
>>>"--use_textid='tru e '"]
>>I think when I came across this error, I added shell=True, e.g.
>>>
>>sub1 = subprocess.Pope n(command, shell=True)

>I added the shell=True and this time it got into Match1 (hurrah!),
>but it then opened up an interactive python session, and didn't
>complete until I manually typed 'exit' in the interactive session.
>>
>Match1 looks like:
>>
>if __name__ == "__main__":
><<< parse arguments >>>
>>
> RunMatch1(file_ ref, file_cmp, iblock_start, iblock_end, \
> nlinep, nsmpp, mindispx, maxdispx, mindispl, \
> maxdispl, istep, chmetric, use_textid)
>>
> exit()
>>
>where the routine RunMatch1 does all the actual processing.
>>
>How do I get Match1 to run and exit normally without opening up an
>interactive session, when called as a subprocess from Match4?
>>

>
Alternately, rather than using a list of arguments, have you tried
just using a string? (Again, that's the way I do it and I haven't
been having any problems recently, although I'm running shell scripts
or binaries with arguments rather than trying to invoke python on a
script.)
>
command = "python ../src_python/Match1.py
--file_ref=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_BF_F03_0024. hdf
--file_cmp=MISR_A M1_GRP_ELLIPSOI D_GM_P228_O0035 71_DF_F03_0024. hdf
--block_start=62 --block_end=62 --istep=16 --chmetric='M2'
--use_textid=true "
>
proc = subprocess.Pope n(command, shell=True)

Thanks - that did the trick. I just passed in one long string
and everything actually works. Wow! I had no idea if this was
even do-able.

This is so cool, and saves me a lot of code duplication. I can
spawn off half a dozen jobs at once and then just wait for them
to finish. It's great that python can function both as a
scripting language and also a full-blown programming language
at the same time.

Catherine