Fetching websites with Python

[Markus Franz]

Hi.

How can I grab websites with a command-line python script? I want to start
the script like this:

../script.py ---xxx--- http://www.address1.com http://www.address2.com

http://www.address3.com

The script should load these 3 websites (or more if specified) in parallel
(may be processes? threads?) and show their contents seperated by ---xxx---.
The whole output should be print on the command-line. Each website should
only have 15 seconds to return the contents (maximum) in order to avoid a
never-ending script.

How can I do this?

Thanks.

Yours sincerely

Markus Franz

[wes weston]

Re: Fetching websites with Python

Markus Franz wrote:[color=blue]
> Hi.
>
> How can I grab websites with a command-line python script? I want to start
> the script like this:
>
> ./script.py ---xxx--- http://www.address1.com http://www.address2.com
> http://www.address3.com
>
> The script should load these 3 websites (or more if specified) in parallel
> (may be processes? threads?) and show their contents seperated by ---xxx---.
> The whole output should be print on the command-line. Each website should
> only have 15 seconds to return the contents (maximum) in order to avoid a
> never-ending script.
>
> How can I do this?
>
> Thanks.
>
> Yours sincerely
>
> Markus Franz
>
>[/color]

Markus,
I think there's a timeout in urllib; not sure.


import urllib
import sys
#--------------------------------------
if __name__ == "__main__":
if len(sys.argv) < 3:
print 'arg error'
sys.exit(1)
sep = sys.argv[1]
for url in sys.argv[2:]:
try:
f = urllib.urlopen( url)
lines = f.readlines()
f.close()
for line in lines:
print line[:-1]
except:
print url,'get error'
print sep

[William Park]

Re: Fetching websites with Python

Markus Franz <mf@orase.com > wrote:[color=blue]
> Hi.
>
> How can I grab websites with a command-line python script? I want to start
> the script like this:
>
> ./script.py ---xxx--- http://www.address1.com http://www.address2.com
> http://www.address3.com
>
> The script should load these 3 websites (or more if specified) in parallel[/color]

In parallel? Hmm... play around with
lynx -dump http://... > a1 &
lynx -dump http://... > a2 &
lynx -dump http://... > a3 &
sleep 15
kill %1 %2 %3
for i in a1 a2 a3; do
cat $i
echo ---xxx---
done
rm a1 a2 a3

In serial, the code becomes
for i in http://... http://... http://... ; do
lynx -connect_timeout =15 -dump $i
echo ---xxx---
done
[color=blue]
> (may be processes? threads?) and show their contents seperated by ---xxx---.
> The whole output should be print on the command-line. Each website should
> only have 15 seconds to return the contents (maximum) in order to avoid a
> never-ending script.
>
> How can I do this?
>
> Thanks.
>
> Yours sincerely
>
> Markus Franz
>
>[/color]

--
William Park, Open Geometry Consulting, <opengeometry@y ahoo.ca>
Linux solution for data processing and document management.

[simo]

Re: Fetching websites with Python

wes weston <wweston@att.ne t> wrote:
[color=blue]
> Markus,
> I think there's a timeout in urllib; not sure.[/color]

No there isn't, bit of a shame that. There is in httplib.

[Technoumena]

Re: Fetching websites with Python

> How can I do this?

Perhaps something like this:

import urllib2, thread, time, sys

thread_count = len(sys.argv) - 1
pages = []
lock = thread.allocate _lock()

def timer():
global lock
time.sleep(15)
lock.release()

def get_page(url):
global thread_count, pages, lock
try: pages.append(ur llib2.urlopen(u rl).read())
except: pass
thread_count -= 1
if thread_count == 0:
lock.release()

lock.acquire()
thread.start_ne w_thread(timer, ())
for url in sys.argv[1:]:
thread.start_ne w_thread(get_pa ge, (url,))
lock.acquire()
print '\n---xxx---\n'.join(pages)



Please have a nice day.

Regards,
Technoumena

[f29]

Re: Fetching websites with Python

> > Markus,[color=blue][color=green]
> > I think there's a timeout in urllib; not sure.[/color]
>
> No there isn't, bit of a shame that. There is in httplib.[/color]

Sure there is, use urllib or urllib2 as usual, but also import socket
module and call "socket.setdefa ulttimeout(secs )" before requesting any
pages with urlopen.

f29

[Andrew Bennetts]

Re: Fetching websites with Python

On Wed, Mar 31, 2004 at 07:33:45PM +0200, Markus Franz wrote:[color=blue]
> Hi.
>
> How can I grab websites with a command-line python script? I want to start
> the script like this:
>
> ./script.py ---xxx--- http://www.address1.com http://www.address2.com
> http://www.address3.com
>
> The script should load these 3 websites (or more if specified) in parallel
> (may be processes? threads?) and show their contents seperated by ---xxx---.
> The whole output should be print on the command-line. Each website should
> only have 15 seconds to return the contents (maximum) in order to avoid a
> never-ending script.
>
> How can I do this?[/color]

You could use Twisted <http://twistedmatrix.c om>:

from twisted.interne t import reactor
from twisted.web.cli ent import getPage
import sys

def gotPage(page):
print seperator
print page

def failed(failure) :
print seperator + ' FAILED'
failure.printTr aceback()

def decrement(ignor ed):
global count
count -= 1
if count == 0:
reactor.stop()

seperator = sys.argv[1]
urlList = sys.argv[2:]
count = len(urlList)
for url in urlList:
getPage(url, timeout=15).add Callbacks(gotPa ge, failed).addBoth (decrement)

reactor.run()

It will grab the sites in parallel, printing them in the order they arrive,
and doesn't use multiple processes, or multiple threads :)

-Andrew.