Deficiency in urllib/socket for https?

**John J. Lee** · Jul 18 '05, 01:53 AM

Re: Deficiency in urllib/socket for https?

Gary Feldman <gafStopSpamDat a@ziplink.stopa llspam.net> writes:
[color=blue]
> I think I've found a deficiency in the design of urllib related to https.
>
> In order to complete an https connection, it appears that URLOpener and
> hence FancyURLOpener require the key and cert files. Or at least, it's not
> clear from the description of socket.ssl what it does if they're omitted.[/color]

Nor from urllib -- see below. In fact, it seems that verification is
just skipped if they're not there.

[color=blue]
> However, urlopen has no way to specify such things. Nor should it - for
> typical uses, a person simply trying to retrieve data from an ssl site
> really doesn't want to know or care about keys and certificate directories.
> One just wants to provide an https url and have it work. Ideally, there
> should be defaults for the certificate files.[/color]

Hmm, looking at both urllib and urllib2, I see urllib2 doesn't use any
key or certificate files at all. So, two points: this is a deficiency
in urllib2 that should be fixed, and, if you're not bothered about key
verification, I'd guess just not providing key / cert files will work.

Hmm, urllib documentation seems wrong here:

Additional keyword parameters, collected in x509, are used for
authentication with the https: scheme. The keywords key_file and
cert_file are supported; both are needed to actually retrieve a
resource at an https: URL.

The fact that https works in urllib2 (which does not provide key /
cert files) seems to demonstrate that they're *not* required, and that
verification is skipped if they're not supplied.

If you *are* bothered about verification, use the x509 arg to
FancyURLOpener (which is documented, see above). The urlopen function
is just a convenience -- just cut-n-paste the trivial code from
urllib.py and adapt it to your needs if you need something more
complicated.

[color=blue]
> This implies that somewhere in the function hierarchy, I suspect in
> socket.ssl, there needs to be some clever defaults. I don't know if they
> folks maintaining the Python distribution really want to be in the business
> of maintaining key and certificate directories (probably not), but there at
> least ought to be a way to specify default directories (oh, no, another
> environment variable?). Thinking idealistically, it would be great if it
> could share the default certs on the system (i.e. on UNIX, find a Netscape
> or Mozilla install directory and use those, and on MS Windows, do whatever
> it takes to use the Windows mechanism).[/color]

That sounds great if you have the time to write the code. Nobody else
is likely to.

John

**John J. Lee** · Jul 18 '05, 01:54 AM

Re: Deficiency in urllib/socket for https?

jjl@pobox.com (John J. Lee) writes:
[...][color=blue]
> Would you mind submitting a doc patch (both urllib and urllib2 docs
> appear to need fixing -- urllib2 to say that it never verifies, urllib
> to say that it skips verification if an appropriate x509 mapping isn't
> supplied)?[/color]

Hmm, maybe I've got this wrong: the fact that key/cert args are passed
to httplib.HTTPS by urllib doesn't mean authentication happens, and
the fact that they're not passed by urllib2 doesn't mean
authentication doesn't happen. I'll investigate.

John

**John J. Lee** · Jul 18 '05, 01:54 AM

Re: Deficiency in urllib/socket for https?

jjl@pobox.com (John J. Lee) writes:
[...][color=blue]
> You're right -- with the caveat that it is useful to have https even
> without authentication (essentially all https traffic on the internet
> proves that ;-).[/color]
[...]

I should have said "...it is useful to have *support* for https...".

The utility of https itself is another matter...

John

**John J. Lee** · Jul 18 '05, 01:54 AM

Re: Deficiency in urllib/socket for https?

jjl@pobox.com (John J. Lee) writes:
[color=blue]
> jjl@pobox.com (John J. Lee) writes:
> [...][color=green]
> > Would you mind submitting a doc patch (both urllib and urllib2 docs
> > appear to need fixing -- urllib2 to say that it never verifies, urllib
> > to say that it skips verification if an appropriate x509 mapping isn't
> > supplied)?[/color]
>
> Hmm, maybe I've got this wrong: the fact that key/cert args are passed
> to httplib.HTTPS by urllib doesn't mean authentication happens, and
> the fact that they're not passed by urllib2 doesn't mean
> authentication doesn't happen. I'll investigate.[/color]

Bah! *After* reading the source, I found this in the ssl module docs:

| Warning: This does not do any certificate verification!

(which the _ssl.c source confirms: it uses SSL_VERIFY_NONE , but
doesn't call SSL_get_verify_ result).

So the urllib docs are wrong:

| Additional keyword parameters, collected in x509, are used for
| authentication with the https: scheme. The keywords key_file and
| cert_file are supported; both are needed to actually retrieve a
| resource at an https: URL.

They're not needed, and they're never used for authentication (if you
don't count just checking the key without verifying it against the
certificate). Given this, the fact that urllib2 doesn't have
arguments for this starts to look like a feature, not a bug! Actually
(dredging up very hazy memories here) aren't you supposed to check a
revocation list, too? Is that given in a URL in the certificate? No
idea how this SSL stuff is supposed to work, really...

I'll upload a doc patch in a minute.

So, in summary, none of httplib, urllib and urllib2 in standard Python
do proper authentication (because the socket module doesn't). There
are third-party SSL libraries for Python: m2crypto is one. If you
need it, and assuming m2crypto has an ssl function with the same
interface that *does* do better auth, I suppose you could probably do

import socket
from m2crypto import ssl # or whatever
socket.ssl = ssl

And have urllib magically start working, with any luck.

John

Deficiency in urllib/socket for https?

Deficiency in urllib/socket for https?

Comment

Comment

Comment

Comment