urlretrieve hangs on large files (sometimes)

[Josh]

Hi. I am writing a script that downloads lots of zips from a usgs site.
All is going well except that occasionally, in the midst of
downloading a file, the script just hangs and i must either ctrl-break
it (running from command line) or, if I wait about 20 minutes after it
hangs, even that won't allow me to close it. Either way, I must then
restart the program and have it pick up where it left off.

I don't know much about python or http, but I believe that the problem
is website sometimes does not respond to the 'get' (or whatever
urlretrieve ) request is in a timely manner: certainly the site
sometimes takes a bit of a break even when downloading with IE, for
example. However IE or other download managers continue -- my python
script just hangs.

Here is the relevant code, I think:

out = file(outFile, "wb")
(filename, headers,) = urllib.urlretri eve(dlinks[count], outFile,
reporthook=repo rt)
out.close()

I am making a haphazard guess that somewhere in the bowels of
urlretrieve there is a timeout setting for the http get command, and if
the site does not respond in time it gets all upset and confused.
Whether I am wrong or right -- any suggestions?

Thanks a lot.

-Josh

[Marcin Ciura]

Re: urlretrieve hangs on large files (sometimes)

Josh wrote:[color=blue]
> Hi. I am writing a script that downloads lots of zips from a usgs site.
> All is going well except that occasionally, in the midst of downloading
> a file, the script just hangs and i must either ctrl-break it (running
> from command line) or, if I wait about 20 minutes after it hangs, even
> that won't allow me to close it. Either way, I must then restart the
> program and have it pick up where it left off.[/color]

Use socket.setdefau lttimeout().
Marcin