looking for wget-like module for getching software

[Robert P. J. Day]

just getting started with python, and i'm designing a program for
fetching software from the net, given the package name, version
number and/or date stamp, download method (tarball, CVS, etc.) and so
on. i've already got a shell script doing this, but python would
certainly clean up the code a lot.

first question -- does such a utility/module (or something like it)
already exist? that is, with the ability to say, "go get me gcc,
version 3.4.3, as a tarball"? or "go get me glibc, CVS date stamp
xxxx, from that CVS repo"?

as for the second question, the driving force behind a fair bit of
the current program is "wget", which has the ability to search
recursively and bring back a file of a given name (which "curl"
doesn't appear to have).

that is, i can just say, "go get file gcc-3.4.2.tar.bz2", and start
searching at "ftp://pub.gnu.org/pub/gcc". i may not know how far down
in the directory structure that file is, but wget will happily search
recursively until it finds it.

so, any pointers to python utils/modules that do any or all of the
above? thanks.

rday

[Stefan Behnel]

Re: looking for wget-like module for getching software


Robert P. J. Day schrieb:[color=blue]
> that is, i can just say, "go get file gcc-3.4.2.tar.bz2", and start
> searching at "ftp://pub.gnu.org/pub/gcc". i may not know how far down
> in the directory structure that file is, but wget will happily search
> recursively until it finds it.[/color]

That sounds pretty inefficient and should produce some load on the server
side. Imagine everyone doing that. Many FTP-servers provide files like
"ls-R.gz" somewhere, meaning, a list of files and directories. And then,
there's Google. Why do you have to use FTP? Ask Google for
"yourfile.tar.b z2 site:the.domain .name". It will very likely return a
suitable URL for you that you can use for download. Just tried it with
"site:ftp.gnu.o rg gcc-3.4.2.tar.bz2" - works. You can even use that link
and try to replace "http://" with "ftp://" for download. Should work for
many source repositories.

You can use urllib2 for both querying and download.

Stefan

[Robert P. J. Day]

Re: looking for wget-like module for getching software

On Thu, 16 Dec 2004, Stefan Behnel wrote:
[color=blue]
>
> Robert P. J. Day schrieb:[color=green]
> > that is, i can just say, "go get file gcc-3.4.2.tar.bz2", and start
> > searching at "ftp://pub.gnu.org/pub/gcc". i may not know how far down
> > in the directory structure that file is, but wget will happily search
> > recursively until it finds it.[/color]
>
> That sounds pretty inefficient and should produce some load on the
> server side. Imagine everyone doing that. Many FTP-servers provide
> files like "ls-R.gz" somewhere, meaning, a list of files and
> directories.[/color]

i realize that and, as much as possible, i try to start the process
with what i *know* is the actual directory location. but
occasionally, if i can't guarantee an exact directory, i just want the
option to start a bit higher up the directory structure. i wasn't
going to make a habit of it, it's more an absolute last resort, if it
comes down to it.
[color=blue]
> And then, there's Google. Why do you have to use FTP? Ask Google for
> "yourfile.tar.b z2 site:the.domain .name". It will very likely return
> a suitable URL for you that you can use for download. Just tried it
> with "site:ftp.gnu.o rg gcc-3.4.2.tar.bz2" - works. You can even use
> that link and try to replace "http://" with "ftp://" for download.
> Should work for many source repositories.
>
> You can use urllib2 for both querying and download.[/color]

and that pretty much answers the question i had. much thanks. and
now that i have my answer, i might have to unsubscribe. you guys sure
are a chatty bunch. :)

rday

p.s. just kidding.