urllib2 through basic auth'ed proxy

[Alejandro Dubrovsky]

I see from googling around that this is a popular topic, but I haven't seen
anyone saying "ah, yes, that works", so here it goes.

How does one connect through a proxy which requires basic authorisation?
The following code, stolen from somewhere, fails with a 407:

proxy_handler = urllib2.ProxyHa ndler({"http" :
"http://the.proxy.addre ss:3128"})
proxy_auth_hand ler = urllib2.ProxyBa sicAuthHandler( )
proxy_auth_hand ler.add_passwor d("The name of the realm sniffed from
telnetting to the proxy and doing a
get",'the.proxy .address','theu sername','thepa ssword')
opener = urllib2.build_o pener(proxy_han dler, proxy_auth_hand ler)
urllib2.install _opener(opener)
f = urllib2.urlopen ('http://www.google.com/')


I still get a 407 if I set the realm to None, I change host to the
'http://the.proxy.addre ss/' form or even 'http://the.proxy.addre ss:3128'
form.

The proxy is squid. Python version is 2.3.4 (I read that this version has a
problem in that it introduces an extra return after the authorisation, but
it isn't even getting to that bit). And yes, going through firefox,
everything works fine.

Can anyone explain me why this fails, or more importantly, code that would
work?

Thanks,
alejandro

[John J. Lee]

Re: urllib2 through basic auth'ed proxy

Alejandro Dubrovsky <dubrovsky@phys ics.uq.edu.au> writes:
[...][color=blue]
> How does one connect through a proxy which requires basic authorisation?
> The following code, stolen from somewhere, fails with a 407:
>[/color]
[...code involving urllib2.ProxyBa sicAuthHandler( )...][color=blue]
> Can anyone explain me why this fails, or more importantly, code that would
> work?[/color]

OK, I finally installed squid and had a look at the urllib2 proxy
basic auth support (which I've steered clear of for years despite
doing quite a bit with urllib2). Seems quite broken. Appears to have
been broken back in December 2004, with revision 38092 (note there's a
little revision number oddness in the Python SVN repo, BTW:

http://mail.python.org/pipermail/python-dev/2005-November/058269.html):

--- urllib2.py (revision 38091)
+++ urllib2.py (revision 38092)
@@ -720,7 +720,10 @@
return self.retry_http _basic_auth(hos t, req, realm)

def retry_http_basi c_auth(self, host, req, realm):
- user,pw = self.passwd.fin d_user_password (realm, host)
+ # TODO(jhylton): Remove the host argument? It depends on whether
+ # retry_http_basi c_auth() is consider part of the public API.
+ # It probably is.
+ user, pw = self.passwd.fin d_user_password (realm, req.get_full_ur l())
if pw is not None:
raw = "%s:%s" % (user, pw)
....


That can't be right, can it? With a proxy, you're always
authenticating yourself for the whole proxy, and you want to look up
(RFC 2617 section 3.2.1). The ProxyBasicAuthH andler subclass
dutifully passes in the right thing for the host argument, but
AbstractBasicAu thHandler ignores it, which means that it never finds
the password -- e.g. if you're trying to connect to python.org through
myproxy.com, it'll be looking for a username/password for python.org
instead of the needed myproxy.com.

Obviously nobody else uses authenticating proxies either, or at least
nobody who can be bothered to fix urllib2 :-(

A workaround is to supply a stupid HTTPPasswordMgr that always returns
the proxy credentials regardless of what the handler asks it for (only
tested with a perhaps-broken 2.5 install, since I've broken my 2.4
install):

import urllib2

class DumbProxyPasswo rdMgr:
def __init__(self):
self.user = self.passwd = None
def add_password(se lf, realm, uri, user, passwd):
self.user = user
self.passwd = passwd
def find_user_passw ord(self, realm, authuri):
return self.user, self.passwd
proxy_auth_hand ler = urllib2.ProxyBa sicAuthHandler( DumbProxyPasswo rdMgr())
proxy_handler = urllib2.ProxyHa ndler({"http": "http://localhost:3128" })
proxy_auth_hand ler.add_passwor d(None, None, 'john', 'blah')
opener = urllib2.build_o pener(proxy_han dler, proxy_auth_hand ler)
f = opener.open('ht tp://python.org/')
print f.read()


Yuck, yuck, yuck! I had realised the auth/proxies code in urllib2 was
buggy, but... And all those hoops to jump through.

Also, if you're using 2.5 SVN HEAD, it seems revision 42133 broke
ProxyHandler in an attempt to fix the URL host:post syntax!

I'll try to get some fixes in tomorrow so that 2.5 isn't broken (or at
least flag the issues to let somebody else fix them), but no promises
as usual...


John

[John J. Lee]

Re: urllib2 through basic auth'ed proxy

Alejandro Dubrovsky <dubrovsky@phys ics.uq.edu.au> writes:
[...][color=blue]
> The proxy is squid. Python version is 2.3.4 (I read that this version has a
> problem in that it introduces an extra return after the authorisation, but
> it isn't even getting to that bit). And yes, going through firefox,
> everything works fine.[/color]
[...]

FWIW, at a glance, Python 2.3.4 has neither of the bugs I mentioned,
but the code I posted seems to work with 2.3.4. I'm not particularly
interested in what's wrong with 2.3.4's version or your usage of it
(probably both), since bugfix releases for 2.3 are no longer
happening, I believe.

I think the Examples section of the docs on this are wrong too, though
that's a bit of a moot point when the code is as broken as it seems...


John

[Alejandro Dubrovsky]

Re: urllib2 through basic auth'ed proxy

John J. Lee wrote:
[color=blue]
> FWIW, at a glance, Python 2.3.4 has neither of the bugs I mentioned,
> but the code I posted seems to work with 2.3.4. I'm not particularly
> interested in what's wrong with 2.3.4's version or your usage of it
> (probably both), since bugfix releases for 2.3 are no longer
> happening, I believe.
>[/color]

"ah, yes, that works" on 2.3.4. Excellent. (I don't see what's so ugly
about that code, but i'm mostly accustomed to my own)
Thanks,
alejandro

[John J. Lee]

Re: urllib2 through basic auth'ed proxy

Alejandro Dubrovsky <dubrovsky@phys ics.uq.edu.au> writes:
[color=blue]
> John J. Lee wrote:
>[color=green]
> > FWIW, at a glance, Python 2.3.4 has neither of the bugs I mentioned,
> > but the code I posted seems to work with 2.3.4. I'm not particularly
> > interested in what's wrong with 2.3.4's version or your usage of it
> > (probably both), since bugfix releases for 2.3 are no longer
> > happening, I believe.
> >[/color]
>
> "ah, yes, that works" on 2.3.4. Excellent. (I don't see what's so ugly
> about that code, but i'm mostly accustomed to my own)[/color]

supplying a password surely shouldn't be that complicated...


John

[John J. Lee]

Re: urllib2 through basic auth'ed proxy

jjl@pobox.com (John J. Lee) writes:
[color=blue]
> Alejandro Dubrovsky <dubrovsky@phys ics.uq.edu.au> writes:[/color]
[...Alejandro complains about non-working HTTP proxy auth in urllib2...]

[...John notes urllib2 bug...][color=blue]
> A workaround is to supply a stupid HTTPPasswordMgr that always returns
> the proxy credentials regardless of what the handler asks it for (only
> tested with a perhaps-broken 2.5 install, since I've broken my 2.4
> install):[/color]
[...snip ugly code][color=blue]
> Yuck, yuck, yuck! I had realised the auth/proxies code in urllib2 was
> buggy, but... And all those hoops to jump through.
>
> Also, if you're using 2.5 SVN HEAD, it seems revision 42133 broke
> ProxyHandler in an attempt to fix the URL host:post syntax![/color]
[...]

In fact the following also works with Python 2.3.4:

import urllib2
proxy_handler = urllib2.ProxyHa ndler({"http": "http://john:blah@local host:3128"})
print urllib2.build_o pener(proxy_han dler).open('htt p://python.org/').read()


....but only just barely skirts around the bugs!-) :-(

(at least, the current bugs: I've no reason to work out what things
were like back in 2.3.4, but the above certainly works with that
version)


John

[Alejandro Dubrovsky]

Re: urllib2 through basic auth'ed proxy

John J. Lee wrote:
[color=blue]
> jjl@pobox.com (John J. Lee) writes:
>[color=green]
>> Alejandro Dubrovsky <dubrovsky@phys ics.uq.edu.au> writes:[/color]
> [...Alejandro complains about non-working HTTP proxy auth in urllib2...]
>
> [...John notes urllib2 bug...][color=green]
>> A workaround is to supply a stupid HTTPPasswordMgr that always returns
>> the proxy credentials regardless of what the handler asks it for (only
>> tested with a perhaps-broken 2.5 install, since I've broken my 2.4
>> install):[/color]
> [...snip ugly code][color=green]
>> Yuck, yuck, yuck! I had realised the auth/proxies code in urllib2 was
>> buggy, but... And all those hoops to jump through.
>>
>> Also, if you're using 2.5 SVN HEAD, it seems revision 42133 broke
>> ProxyHandler in an attempt to fix the URL host:post syntax![/color]
> [...]
>
> In fact the following also works with Python 2.3.4:
>
> import urllib2
> proxy_handler = urllib2.ProxyHa ndler({"http":
> "http://john:blah@local host:3128"}) print
> urllib2.build_o pener(proxy_han dler).open('htt p://python.org/').read()
>[/color]
It does too. Thanks again. (I think this version is uglier, but easier to
insert into third party code)