FTP with urllib2 behind a proxy

Re: FTP with urllib2 behind a proxy

You need to install a proxyhandler, authhandler,
ftphandler and httphandler. Then build yourself
an opener, that opens the doors for you ... :-)

The following does the trick.

proxy_handler = urllib2.ProxyHa ndler( {'http': 'myhttpproxy:80 ',
'https' : 'myhttpsproxy:4 43',
'ftp' : 'myftpproxy:21' } )

opener= urllib2.build_o pener(proxy_han dler, urllib2.HTTPBas icAuthHandler() ,
urllib2.HTTPHan dler, urllib2.HTTPSHa ndler,
urllib2.FTPHand ler)

# install this opener
urllib2.install _opener(opener)

# Go ahead, knock knock!

req=urlli2.Requ est('ftp://ftp.gnu.org')
data=urllib2.ur lopen(req).read ()

Of course, replace the arbit proxy values I wrote
with your proxy values. If your proxy need authentication
, you will need to do a bit more here.

proxyauth='http ://' + username + '@' + password + 'myproxy:myprox port'

Then your proxy handler becomes ( I am assuming a generic proxy
for all protocols here!)

proxy_handler = urllib2.ProxyHa ndler ( {'http' : proxyauth,
'https' : proxyauth,
'ftp' : proxyauth } )

HTH.

-Anand


jjl@pobox.com (John J. Lee) wrote in message news:<87d6feykl 8.fsf@pobox.com >...[color=blue]
> Zappelphillip@g mx.de (O. Koch) writes:
>[color=green]
> > Until now, i know that ftplib doesn't support proxies and that i have
> > to use urllib2. But i don't know how to use the urllib2 correct. I
> > found some examples, but i don't understand them.
> >
> > Is there anyone who can help me?[/color]
>
> import urllib2
> response = urllib2.urlopen ("ftp://ftp.example.com/pub/myfile")
> data = response.read()
> response.close( )
>
>
> Does that do the trick?
>
>
> John[/color]

Re: FTP with urllib2 behind a proxy

pythonguy@Hotpo p.com (Anand Pillai) writes:
[color=blue]
> You need to install a proxyhandler, authhandler,
> ftphandler and httphandler. Then build yourself
> an opener, that opens the doors for you ... :-)
>
> The following does the trick.
>
> proxy_handler = urllib2.ProxyHa ndler( {'http': 'myhttpproxy:80 ',
> 'https' : 'myhttpsproxy:4 43',
> 'ftp' : 'myftpproxy:21' } )
>
> opener= urllib2.build_o pener(proxy_han dler, urllib2.HTTPBas icAuthHandler() ,
> urllib2.HTTPHan dler, urllib2.HTTPSHa ndler,
> urllib2.FTPHand ler)
>
> # install this opener
> urllib2.install _opener(opener)
>
> # Go ahead, knock knock!
>
> req=urlli2.Requ est('ftp://ftp.gnu.org')
> data=urllib2.ur lopen(req).read ()[/color]

A couple of things to add: you don't need to add handlers that already
get added by default by build_opener (FTPHandler and HTTPHandler, for
example). ProxyHandler is one of these default handlers, so if your
environment is set up for it (http_proxy, etc. environment variables),
you don't need to supply a ProxyHandler (of course, if your
environ. *isn't*, then you do need to supply one, to give it the proxy
details). You don't need Request objects (unless you want to add
headers to a Request, or pass Requests around). You don't need to
install a global opener, unless your code expects it -- it's just a
convenience (or an inconvenience, sometimes).

Actually, did the OP say proxy basic auth. was involved? Don't
recall. I've never needed it for proxies, but there seems to be a
ProxyBasicAuthH andler in urllib2, so I guess that's what you meant to
use, rather than HTTPBasicAuthHa ndler (which is for website auth., not
proxy auth).

So, after all that, you end up with:

opener = urllib2.build_o pener(urllib2.P roxyBasicAuthHa ndler)
data = opener.open('ft p://ftp.gnu.org').r ead()

(I like to close the response explicitly, though)


John

Re: FTP with urllib2 behind a proxy

jjl@pobox.com (John J. Lee) wrote in message news:<87wudhoh5 c.fsf@pobox.com >...[color=blue]
> pythonguy@Hotpo p.com (Anand Pillai) writes:[/color]
The code obviously caters to the bottom-line, in case
the programmer does not want to worry about environment
variables, would like to install the handlers himself
and his proxy needs authentication.

In the last case, the HTTP_PROXY env variable does not
help and you need to install the proxy handler yourself.

I copied this stuff from my program which needs all these
and more ( a USER-AGENT header for example ).

-Anand
[color=blue][color=green]
> > You need to install a proxyhandler, authhandler,
> > ftphandler and httphandler. Then build yourself
> > an opener, that opens the doors for you ... :-)
> >
> > The following does the trick.
> >
> > proxy_handler = urllib2.ProxyHa ndler( {'http': 'myhttpproxy:80 ',
> > 'https' : 'myhttpsproxy:4 43',
> > 'ftp' : 'myftpproxy:21' } )
> >
> > opener= urllib2.build_o pener(proxy_han dler, urllib2.HTTPBas icAuthHandler() ,
> > urllib2.HTTPHan dler, urllib2.HTTPSHa ndler,
> > urllib2.FTPHand ler)
> >
> > # install this opener
> > urllib2.install _opener(opener)
> >
> > # Go ahead, knock knock!
> >
> > req=urlli2.Requ est('ftp://ftp.gnu.org')
> > data=urllib2.ur lopen(req).read ()[/color]
>
> A couple of things to add: you don't need to add handlers that already
> get added by default by build_opener (FTPHandler and HTTPHandler, for
> example). ProxyHandler is one of these default handlers, so if your
> environment is set up for it (http_proxy, etc. environment variables),
> you don't need to supply a ProxyHandler (of course, if your
> environ. *isn't*, then you do need to supply one, to give it the proxy
> details). You don't need Request objects (unless you want to add
> headers to a Request, or pass Requests around). You don't need to
> install a global opener, unless your code expects it -- it's just a
> convenience (or an inconvenience, sometimes).
>
> Actually, did the OP say proxy basic auth. was involved? Don't
> recall. I've never needed it for proxies, but there seems to be a
> ProxyBasicAuthH andler in urllib2, so I guess that's what you meant to
> use, rather than HTTPBasicAuthHa ndler (which is for website auth., not
> proxy auth).
>
> So, after all that, you end up with:
>
> opener = urllib2.build_o pener(urllib2.P roxyBasicAuthHa ndler)
> data = opener.open('ft p://ftp.gnu.org').r ead()
>
> (I like to close the response explicitly, though)
>
>
> John[/color]

Re: FTP with urllib2 behind a proxy

pythonguy@Hotpo p.com (Anand Pillai) writes:
[color=blue]
> jjl@pobox.com (John J. Lee) wrote in message news:<87wudhoh5 c.fsf@pobox.com >...[color=green]
> > pythonguy@Hotpo p.com (Anand Pillai) writes:[/color]
> The code obviously caters to the bottom-line, in case
> the programmer does not want to worry about environment
> variables, would like to install the handlers himself
> and his proxy needs authentication.[/color]

Sure, I understand that. I was just elaborating, no criticism was
intended!

It seems people are frightened to use urllib2 (perhaps partly because
the documentation has lots of sections -- though they're all very
short), so I like to point out that most of the time it's just a
matter of:

import urllib2
urllib2.urlopen ("http://example.com/")


And most of the rest of the time, the only complications above that
are passing a Request instead of a URL (for adding headers and passing
around requests as first-class objects), and building your own opener
with build_opener (for choosing what features you want and don't want,
like authentication -- and adding your own features, of course).


John