Win XP: Problem with shell scripting in Python

[A.M]

Hi,



I am having difficulty with shell scripting in Python.



I use the following command to run a DOS command and put the return value in
a Python variable:



print os.popen('DIR') .read()



It works very fine with DIR command, but for commands like "MD :" it doesn't
return the error message into the string:



print os.popen('MD :').read()

# No error message



When I use Ruby, it works perfect:



`md :`

The filename, directory name, or volume label syntax is incorrect.



I am also having problem with redirecting the python script output to a
file: That means I can redirect the output to a file by using pipes like
this:



Python.exe script.py >file.txt



But the sequence of contents in file.txt doesn't match with command
execution sequence!

When I don't use pipes, the output sequence is fine when I see the output on
the monitor screen.



Am I missing anything? Considering the fact that Ruby doesn't have any
problem with redirecting STDOUT into files or string variables, is Python
the right tool for this kinds of shell scripting?



Any help would be appreciated,

Alan

[John Machin]

Re: Win XP: Problem with shell scripting in Python

On 9/06/2006 8:58 AM, A.M wrote:[color=blue]
> Hi,
> I am having difficulty with shell scripting in Python.
> I use the following command to run a DOS command and put the return value in
> a Python variable:
> print os.popen('DIR') .read()
> It works very fine with DIR command, but for commands like "MD :" it doesn't
> return the error message into the string:
> print os.popen('MD :').read()
> # No error message
> When I use Ruby, it works perfect:
> `md :`
>[/color]

Irrelevant; different "it".
[color=blue]
> The filename, directory name, or volume label syntax is incorrect.
>
>[/color]

I have never had occasion to use os.popen_anythi ng before. Perhaps we
can aid each other on the path to enlightenment. I got the following
idea from looking at the manual.

|>>> handles = os.popen3('MD :')
|>>> [f.read() for f in handles[1:]]
['', 'The filename, directory name, or volume label syntax is incorrect.\n']
|>>> [f.close() for f in handles]
[None, None, 1]

|>>> handles = os.popen3('dir *.py')
|>>> [f.read() for f in handles[1:]]
[' Volume in drive C has no label.\n Volume Serial Number is **BIG
SNIP** bytes free\n', '']
|>>> [f.close() for f in handles]
[None, None, None]

Windows does occasionally adhere to *x conventions like "data to stdout,
error messages to stderr" :-)

Now it's *your* turn to do something for the cause. It appears to me
that popen4 has exactly the same documentation as popen3, as recently as
2.5a2. I see no fourth gizmoid here.

Whoooaaah! "4" is not a gizmoid count:

|>>> handles = os.popen4('MD :')
|>>> handles
(<open file 'MD :', mode 'w' at 0x00AAF698>, <open file 'MD :', mode 'r'
at 0x00
AAF4E8>)

You might like to suss this out, and raise a request to have the docs fixed.

Cheers,
John

[John Machin]

Re: Win XP: Problem with shell scripting in Python

On 9/06/2006 10:47 AM, John Machin wrote:
[color=blue]
> Now it's *your* turn to do something for the cause. It appears to me
> that popen4 has exactly the same documentation as popen3, as recently as
> 2.5a2. I see no fourth gizmoid here.
>
> Whoooaaah! "4" is not a gizmoid count:
>
> |>>> handles = os.popen4('MD :')
> |>>> handles
> (<open file 'MD :', mode 'w' at 0x00AAF698>, <open file 'MD :', mode 'r'
> at 0x00
> AAF4E8>)
>
> You might like to suss this out, and raise a request to have the docs
> fixed.[/color]

OK, OK, alright already. I didn't read the docs closely enough. Forget
the doc-fix request. It looks like it will pay you to investigate popen4
a bit further :-)

Cheers,

[Fredrik Lundh]

Re: Win XP: Problem with shell scripting in Python

A.M wrote:
[color=blue]
> It works very fine with DIR command, but for commands like "MD :" it doesn't
> return the error message into the string:
>
> print os.popen('MD :').read()
>
> # No error message[/color]

in python, "MD" is spelled os.mkdir.
[color=blue]
> Am I missing anything?[/color]

the difference between STDOUT and STDERR, and the difference between
buffered output and non-buffered output, and perhaps a few other things
related to how STDIO behaves on modern computers... however, if you
want to pretend that STDOUT and STDERR are the same thing, you can use
os.popen4:
[color=blue][color=green][color=darkred]
>>> o, i = os.popen4("md :")
>>> i.read()[/color][/color][/color]
'The filename, directory name, or volume label syntax is incorrect.\n'

or the subprocess module.
[color=blue]
> Considering the fact that Ruby doesn't have any problem with redirecting
> STDOUT into files or string variables, is Python the right tool for
> this kinds of shell scripting?[/color]

rewriting BAT files as a series of os.system or os.popen calls isn't
exactly optimal (neither for the computer nor the programmer nor the
future user); better take an hour to skim the "generic operating system
services" section in the library reference, and use built-in functions
wherever you can:

http://docs.python.org/lib/allos.html

the following modules are especially useful:

http://docs.python.org/lib/module-os.html

http://docs.python.org/lib/module-os.path.html

http://docs.python.org/lib/module-glob.html

http://docs.python.org/lib/module-shutil.html

by using the built-in tools, you get better performance in many cases,
better error handling, and code that's a lot easier to reuse (also on
non-Windows platforms).

</F>

[A.M]

Re: Win XP: Problem with shell scripting in Python


"Fredrik Lundh" <fredrik@python ware.com> wrote in message
news:mailman.67 85.1149837599.2 7775.python-list@python.org ...[color=blue]
> A.M wrote:
>
>
> in python, "MD" is spelled os.mkdir.
>[color=green]
>> Am I missing anything?[/color]
>
> the difference between STDOUT and STDERR, and the difference between
> buffered output and non-buffered output, and perhaps a few other things
> related to how STDIO behaves on modern computers... however, if you want
> to pretend that STDOUT and STDERR are the same thing, you can use
> os.popen4:
>[color=green][color=darkred]
> >>> o, i = os.popen4("md :")
> >>> i.read()[/color][/color]
> 'The filename, directory name, or volume label syntax is incorrect.\n'
>
> or the subprocess module.
>[color=green]
>> Considering the fact that Ruby doesn't have any problem with redirecting
> > STDOUT into files or string variables, is Python the right tool for
> > this kinds of shell scripting?[/color]
>
> rewriting BAT files as a series of os.system or os.popen calls isn't
> exactly optimal (neither for the computer nor the programmer nor the
> future user); better take an hour to skim the "generic operating system
> services" section in the library reference, and use built-in functions
> wherever you can:
>
> http://docs.python.org/lib/allos.html
>
> the following modules are especially useful:
>
> http://docs.python.org/lib/module-os.html
> http://docs.python.org/lib/module-os.path.html
> http://docs.python.org/lib/module-glob.html
> http://docs.python.org/lib/module-shutil.html
>
> by using the built-in tools, you get better performance in many cases,
> better error handling, and code that's a lot easier to reuse (also on
> non-Windows platforms).
>
> </F>
>[/color]


Thanks Fredrik for help.



The "MD :" is just a sample. The actual script contains different commands.



The actual script that I am "translatin g" consolidates huge table data from
multiple SQL Server database into Oracle. I have to use BCP command line at
the SQL server side and SQL*Loader at the Oracle side.



I must capture the stdout/stderr output of command lines into log files for
future inspection/troubleshooting . Beside the issue with stdout/stderror,
the main stressful problem that I have is the fact that Python captures
command line's output somehow differently. For example, popen captures BCP's
command output completely wrong. Some part of summary is at the top and the
progress percentages are at the bottom and more.! This is just stdout
output.



I am going to investigate other popen4 and other popen forms per your
suggestion and try to fix the stdout sequence problem.



Regards,

Alan

[Steve Holden]

Re: Win XP: Problem with shell scripting in Python

A.M wrote:[color=blue]
> "Fredrik Lundh" <fredrik@python ware.com> wrote in message
> news:mailman.67 85.1149837599.2 7775.python-list@python.org ...
>[color=green]
>>A.M wrote:
>>
>>
>>in python, "MD" is spelled os.mkdir.
>>
>>[color=darkred]
>>>Am I missing anything?[/color]
>>
>>the difference between STDOUT and STDERR, and the difference between
>>buffered output and non-buffered output, and perhaps a few other things
>>related to how STDIO behaves on modern computers... however, if you want
>>to pretend that STDOUT and STDERR are the same thing, you can use
>>os.popen4:
>>
>>[color=darkred]
>>>>>o, i = os.popen4("md :")
>>>>>i.read()[/color]
>>
>>'The filename, directory name, or volume label syntax is incorrect.\n'
>>
>>or the subprocess module.
>>
>>[color=darkred]
>>>Considerin g the fact that Ruby doesn't have any problem with redirecting
>>>STDOUT into files or string variables, is Python the right tool for
>>>this kinds of shell scripting?[/color]
>>
>>rewriting BAT files as a series of os.system or os.popen calls isn't
>>exactly optimal (neither for the computer nor the programmer nor the
>>future user); better take an hour to skim the "generic operating system
>>services" section in the library reference, and use built-in functions
>>wherever you can:
>>
>> http://docs.python.org/lib/allos.html
>>
>>the following modules are especially useful:
>>
>> http://docs.python.org/lib/module-os.html
>> http://docs.python.org/lib/module-os.path.html
>> http://docs.python.org/lib/module-glob.html
>> http://docs.python.org/lib/module-shutil.html
>>
>>by using the built-in tools, you get better performance in many cases,
>>better error handling, and code that's a lot easier to reuse (also on
>>non-Windows platforms).
>>
>></F>
>>[/color]
>
>
> Thanks Fredrik for help.
>
>
>
> The "MD :" is just a sample. The actual script contains different commands.
>
>
>
> The actual script that I am "translatin g" consolidates huge table data from
> multiple SQL Server database into Oracle. I have to use BCP command line at
> the SQL server side and SQL*Loader at the Oracle side.
>
>
>
> I must capture the stdout/stderr output of command lines into log files for
> future inspection/troubleshooting . Beside the issue with stdout/stderror,
> the main stressful problem that I have is the fact that Python captures
> command line's output somehow differently. For example, popen captures BCP's
> command output completely wrong. Some part of summary is at the top and the
> progress percentages are at the bottom and more.! This is just stdout
> output.
>
>
>
> I am going to investigate other popen4 and other popen forms per your
> suggestion and try to fix the stdout sequence problem.
>[/color]
I dare hardly suggest this, but might it not be better to use Python's
database functionality to perform the task? The language can access both
databases, and you might find it quicker. Then again, if your database
experience is limited, you may not ...

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Love me, love my blog http://holdenweb.blogspot.com
Recent Ramblings http://del.icio.us/steve.holden

[A.M]

Re: Win XP: Problem with shell scripting in Python

[color=blue]
> I dare hardly suggest this, but might it not be better to use Python's
> database functionality to perform the task? The language can access both
> databases, and you might find it quicker. Then again, if your database
> experience is limited, you may not ...
>
> regards
> Steve
> --
> Steve Holden +44 150 684 7255 +1 800 494 3119
> Holden Web LLC/Ltd http://www.holdenweb.com
> Love me, love my blog http://holdenweb.blogspot.com
> Recent Ramblings http://del.icio.us/steve.holden
>[/color]

Hi Steven,



Based on my experience, the fastest possible way to import raw data into
Oracle is SQL*Loader. Similarly, the fastest way to extract raw data from
SQL server is BCP.



My script transfers 40,000,000 records (actually big records) from sql
server to oracle in 20 Min. I tried ODBC to do the same work. I turned off
all record locking and transactions through query hints. The actual program
was a C# program. After 12 hours, I just stopped the program.



I created DOS batch files to control BCP and SQL*Loader steps. It is faster
than any fancy GUI tools. Now I am using Python.



I am thinking to add comprehensive logging to the ETL (extract transform
load) process. All details command line outputs will be stored in database.
System administrators can query database and watch how the ETL job is
working.



At this point I have quite challenge with capturing BCP's stdout/stderr
output to string variables in Python program.



I'll post the final outcome here.



Regards, Alan

[Steve Holden]

Re: Win XP: Problem with shell scripting in Python

A.M wrote:[color=blue][color=green]
>>I dare hardly suggest this, but might it not be better to use Python's
>>database functionality to perform the task? The language can access both
>>databases, and you might find it quicker. Then again, if your database
>>experience is limited, you may not ...
>>
>>regards
>> Steve
>>--
>>Steve Holden +44 150 684 7255 +1 800 494 3119
>>Holden Web LLC/Ltd http://www.holdenweb.com
>>Love me, love my blog http://holdenweb.blogspot.com
>>Recent Ramblings http://del.icio.us/steve.holden
>>[/color]
>
>
> Hi Steven,
>
>
>
> Based on my experience, the fastest possible way to import raw data into
> Oracle is SQL*Loader. Similarly, the fastest way to extract raw data from
> SQL server is BCP.
>
>
>
> My script transfers 40,000,000 records (actually big records) from sql
> server to oracle in 20 Min. I tried ODBC to do the same work. I turned off
> all record locking and transactions through query hints. The actual program
> was a C# program. After 12 hours, I just stopped the program.
>[/color]
I'm not that surprised - it was just an inquiry. Usually the database
bulk dump and load utilities take advantage of every trick in their
respective books to provide speed. A native driver might have done
better than ODBC, but you are probably correct on going that route for
speed.[color=blue]
>
>
> I created DOS batch files to control BCP and SQL*Loader steps. It is faster
> than any fancy GUI tools. Now I am using Python.
>
> I am thinking to add comprehensive logging to the ETL (extract transform
> load) process. All details command line outputs will be stored in database.
> System administrators can query database and watch how the ETL job is
> working.
>
> At this point I have quite challenge with capturing BCP's stdout/stderr
> output to string variables in Python program.
>
> I'll post the final outcome here.
>[/color]
OK, good luck.

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Love me, love my blog http://holdenweb.blogspot.com
Recent Ramblings http://del.icio.us/steve.holden

[A.M]

Re: Win XP: Problem with shell scripting in Python



Here is what I came up with after John and Fredrik's help.



import os

import sys



def Execute(shell_c ommand,logStrea m = sys.stdout):

print >>logStream, shell_command

child_stdin, child_stdout_an d_stderr = os.popen4(shell _command)

commad_output = child_stdout_an d_stderr.read()

print >>logStream, commad_output

return_code = child_stdout_an d_stderr.close( )

return_code = return_code or child_stdin.clo se()

print >>logStream, "Return Code: " , return_code



Execute ("DIR")

Execute ("MD :")



I tested it and so far it behaves the way that I want.



The tricky part is that when you use popen4, you have to close both returned
streams to be able to get the return code. I wasn't able to find that in the
documentation.



Alan

[John Machin]

Re: Win XP: Problem with shell scripting in Python

On 10/06/2006 3:00 AM, A.M wrote:[color=blue]
> Here is what I came up with after John and Fredrik's help.
>
> import os
> import sys
> def Execute(shell_c ommand,logStrea m = sys.stdout):
> print >>logStream, shell_command
> child_stdin, child_stdout_an d_stderr = os.popen4(shell _command)
> commad_output = child_stdout_an d_stderr.read()
> print >>logStream, commad_output
> return_code = child_stdout_an d_stderr.close( )
> return_code = return_code or child_stdin.clo se()
> print >>logStream, "Return Code: " , return_code
>
> Execute ("DIR")
>
> Execute ("MD :")
>
> I tested it and so far it behaves the way that I want.
>[/color]

Does it overcome the problem that you reported earlier, that the
contents of the output file from BCP were out of order? If not, you may
like to try popen3(). It's quite possible (and indeed desirable) that
the child's stderr is not buffered (so that error messages appear
immediately) but the child's stdout is buffered (for efficiency), and
when the buffer is flushed governs the order of appearance in a single
output stream.[color=blue]
>
>
> The tricky part is that when you use popen4, you have to close both returned
> streams to be able to get the return code. I wasn't able to find that in the
> documentation.[/color]

In general it is good practice to hand back resources (e.g. close files)
explicitly as soon as you are finished with them. This is especially
important for files open for writing, in case there are problems like
out of disk space, device not functioning etc. Also when you are dealing
with a child process it makes some sense to close its stdin first just
in case it is waiting for that, and will then write something to stdout,
which may fail, causing it to write to stderr. So I guess that the
documenter didn't stumble onto the "tricky part" :-)

The "tricky part" for popen and friends seems to be that the return code
is handed back upon close of the *last* file:

|>>> h = os.popen3('md : ')
|>>> [h[x].close() for x in 2, 1, 0]
[None, None, 1]

Looks like you get to report a documentation "problem" after all :-)

Cheers,
John

[A.M]

Re: Win XP: Problem with shell scripting in Python

>Does it overcome the problem that you reported earlier, that the
contents of the output file from BCP were out of order?



Yes, it does. But, to be honest, I don't know how!!!





"John Machin" <sjmachin@lexic on.net> wrote in message
news:4489efa0@n ews.eftel.com.. .[color=blue]
> On 10/06/2006 3:00 AM, A.M wrote:[color=green]
>> Here is what I came up with after John and Fredrik's help.
>>
>> import os
>> import sys
>> def Execute(shell_c ommand,logStrea m = sys.stdout):
>> print >>logStream, shell_command
>> child_stdin, child_stdout_an d_stderr = os.popen4(shell _command)
>> commad_output = child_stdout_an d_stderr.read()
>> print >>logStream, commad_output
>> return_code = child_stdout_an d_stderr.close( )
>> return_code = return_code or child_stdin.clo se()
>> print >>logStream, "Return Code: " , return_code
>>
>> Execute ("DIR")
>>
>> Execute ("MD :")
>>
>> I tested it and so far it behaves the way that I want.
>>[/color]
>
> Does it overcome the problem that you reported earlier, that the contents
> of the output file from BCP were out of order? If not, you may like to try
> popen3(). It's quite possible (and indeed desirable) that the child's
> stderr is not buffered (so that error messages appear immediately) but the
> child's stdout is buffered (for efficiency), and when the buffer is
> flushed governs the order of appearance in a single output stream.[color=green]
>>
>>
>> The tricky part is that when you use popen4, you have to close both
>> returned streams to be able to get the return code. I wasn't able to find
>> that in the documentation.[/color]
>
> In general it is good practice to hand back resources (e.g. close files)
> explicitly as soon as you are finished with them. This is especially
> important for files open for writing, in case there are problems like out
> of disk space, device not functioning etc. Also when you are dealing with
> a child process it makes some sense to close its stdin first just in case
> it is waiting for that, and will then write something to stdout, which may
> fail, causing it to write to stderr. So I guess that the documenter didn't
> stumble onto the "tricky part" :-)
>
> The "tricky part" for popen and friends seems to be that the return code
> is handed back upon close of the *last* file:
>
> |>>> h = os.popen3('md : ')
> |>>> [h[x].close() for x in 2, 1, 0]
> [None, None, 1]
>
> Looks like you get to report a documentation "problem" after all :-)
>
> Cheers,
> John
>
>[/color]