Popen and wget, problems

[Jesse]

Hi all, I have a problem using wget and Popen. I hope someone can help.


-- Problem --
I want to use the command:
wget -nv -O "dir/cpan.txt" "http://search.cpan.org "
and capture all it's stdout+stderr.
(Note that option -O requires 'dir' to be existing before wget is executed)

Popen doesn't work, while os.system and shell do. Popen will give the error:
dir/cpan.txt: No such file or directory

While os.system and shell will give the correct result:
06:52:40 URL:http://search.cpan.org/ [3657/3657] -"dir1/cpan.txt" [1]



-- Background info about wget --
-Option -nv: -nv, --no-verbose turn off verboseness, without
being quiet.
-Option -O: -O, --output-document=FILE write documents to FILE.

Note that wget requires any directories in the file-path of option -O to be
existing before the wget command is executed.


-- Python Code using Popen with cmd arg list --
# imports
import os
from subprocess import Popen, PIPE

# vars and create dir
cmd_set = ['wget', '-nv', '-O dir/cpan.txt', 'http://search.span.org ']
cmd = ' '.join(cmd_set)
print "cmd: " + cmd
try:
os.makedirs('di r')
except:
print 'dir already exists'


# execute using Popen (does NOT work)
proc = Popen(cmd_set, stdout=PIPE, stderr=PIPE)
return_code = proc.wait()
if return_code == 0:
print "Success:\n %s" % (proc.stdout.re ad() + proc.stderr.rea d())
else:
print "Failure %s:\n%s" % (return_code, proc.stderr.rea d() +
proc.stdout.rea d())


# execute using os.system (does work)
os.system(cmd)


-- Python code output of Popen --
Failure 1:
dir/cpan.txt: No such file or directory


-- Question --
Why is Popen unable to correctly execute the wget, while os.system can?

[Rob Wolfe]

Re: Popen and wget, problems

"Jesse" <hate@spam.comw rites:

Hi all, I have a problem using wget and Popen. I hope someone can help.
>
>
-- Problem --
I want to use the command:
wget -nv -O "dir/cpan.txt" "http://search.cpan.org "
and capture all it's stdout+stderr.
(Note that option -O requires 'dir' to be existing before wget is executed)
>
Popen doesn't work, while os.system and shell do. Popen will give the error:
dir/cpan.txt: No such file or directory
>
While os.system and shell will give the correct result:
06:52:40 URL:http://search.cpan.org/ [3657/3657] -"dir1/cpan.txt" [1]

[...]

-- Python Code using Popen with cmd arg list --
# imports
import os
from subprocess import Popen, PIPE
>
# vars and create dir
cmd_set = ['wget', '-nv', '-O dir/cpan.txt', 'http://search.span.org ']
cmd = ' '.join(cmd_set)
print "cmd: " + cmd
try:
os.makedirs('di r')
except:
print 'dir already exists'
>
>
# execute using Popen (does NOT work)
proc = Popen(cmd_set, stdout=PIPE, stderr=PIPE)
return_code = proc.wait()
if return_code == 0:
print "Success:\n %s" % (proc.stdout.re ad() + proc.stderr.rea d())
else:
print "Failure %s:\n%s" % (return_code, proc.stderr.rea d() +
proc.stdout.rea d())
>
>
# execute using os.system (does work)
os.system(cmd)
>
>
-- Python code output of Popen --
Failure 1:
dir/cpan.txt: No such file or directory
>
>
-- Question --
Why is Popen unable to correctly execute the wget, while os.system can?

I don't know exactly why in this case Popen doesn't work,
but the counterpart of os.system is Popen with option shell=True
and the first parameter should be a string instead of list.
That seems to work:
proc = Popen("wget -nv -O dir/cpan.txt http://search.span.org ",
shell=True, stdout=PIPE, stderr=PIPE)

and this variant seems to work too:
cmd_set = ['wget', '-nv', '-O', 'dir/cpan.txt', 'http://search.span.org ']

--
HTH,
Rob

[Jesse]

Re: Popen and wget, problems

Thx Rob!

Your solution works perfect!


"Rob Wolfe" <rw@smsnet.plwr ote in message
news:87d515ejil .fsf@merkury.sm snet.pl...

"Jesse" <hate@spam.comw rites:
>

>Hi all, I have a problem using wget and Popen. I hope someone can help.
>>
>>
>-- Problem --
>I want to use the command:
>wget -nv -O "dir/cpan.txt" "http://search.cpan.org "
>and capture all it's stdout+stderr.
>(Note that option -O requires 'dir' to be existing before wget is
>executed)
>>
>Popen doesn't work, while os.system and shell do. Popen will give the
>error:
>dir/cpan.txt: No such file or directory
>>
>While os.system and shell will give the correct result:
>06:52:40 URL:http://search.cpan.org/ [3657/3657] -"dir1/cpan.txt" [1]

>
[...]
>

>-- Python Code using Popen with cmd arg list --
># imports
>import os
>from subprocess import Popen, PIPE
>>
># vars and create dir
>cmd_set = ['wget', '-nv', '-O dir/cpan.txt', 'http://search.span.org ']
>cmd = ' '.join(cmd_set)
>print "cmd: " + cmd
>try:
>os.makedirs('d ir')
>except:
>print 'dir already exists'
>>
>>
># execute using Popen (does NOT work)
>proc = Popen(cmd_set, stdout=PIPE, stderr=PIPE)
>return_code = proc.wait()
>if return_code == 0:
>print "Success:\n %s" % (proc.stdout.re ad() + proc.stderr.rea d())
>else:
>print "Failure %s:\n%s" % (return_code, proc.stderr.rea d() +
>proc.stdout.re ad())
>>
>>
># execute using os.system (does work)
>os.system(cm d)
>>
>>
>-- Python code output of Popen --
>Failure 1:
>dir/cpan.txt: No such file or directory
>>
>>
>-- Question --
>Why is Popen unable to correctly execute the wget, while os.system can?

>
I don't know exactly why in this case Popen doesn't work,
but the counterpart of os.system is Popen with option shell=True
and the first parameter should be a string instead of list.
That seems to work:
proc = Popen("wget -nv -O dir/cpan.txt http://search.span.org ",
shell=True, stdout=PIPE, stderr=PIPE)
>
and this variant seems to work too:
cmd_set = ['wget', '-nv', '-O', 'dir/cpan.txt', 'http://search.span.org ']
>
--
HTH,
Rob

[js]

Re: Popen and wget, problems

Hi Jesse.

cmd_set = ['wget', '-nv', '-O dir/cpan.txt', 'http://search.span.org ']

[snip]

>proc = Popen(cmd_set, stdout=PIPE, stderr=PIPE)

wget will treat this as
$ wget -nv '-O dir/cpan.txt' "http://search.cpan.org "

And will emit the following error because there's no pathname ' dir/cpan.txt'.
(Note the pathname has a trailing space.)

dir/cpan.txt: No such file or directory

Replace '-O dir/cpan.txt" with '-Odir/cpan.txt' or '-O', 'dir/cpan.txt'
and it should work.

Hope this helps