Web authentication

[luigipaioro]

Good morning to all!

I'm trying to access on a web page that needs user and password
authentication. I'm enabled to access there (I mean that I have an
user name and a password to access via web), but I cannot access using
an automatic procedure (that is what I need to make a daemon that
downloads weekly an ASCII file from that site).

I've tried using urllib:

import urllib

conn = urlib.urlopen(" http://user:password@w ww.mysite.com")
print conn.read()

But it doesn't work (it asks me again user and password).

Does anybody know how can I acces to my site with authentication?

I think that urllib2 can help me but I don't undestand how!!!

Thaks

Luigi

[John J. Lee]

Re: Web authentication

"luigipaior o" <luigipaioro@li bero.it> writes:
[color=blue]
> Good morning to all![/color]

Good morning! No need to post twice, BTW.

[color=blue]
> I'm trying to access on a web page that needs user and password
> authentication. I'm enabled to access there (I mean that I have an
> user name and a password to access via web), but I cannot access using
> an automatic procedure (that is what I need to make a daemon that
> downloads weekly an ASCII file from that site).
>
> I've tried using urllib:
>
> import urllib
>
> conn = urlib.urlopen(" http://user:password@w ww.mysite.com")
> print conn.read()
>
> But it doesn't work (it asks me again user and password).[/color]

That URL should work for "Basic HTTP authentiation" using urllib (I
think -- I always use urllib2, so not certain about urllib). For some
reason, a quick glance at the code suggests it *won't* work with
urllib2, but it's easy enough to achieve the same result with that
module (see the link below for how). The page you're accessing may
need some other means of authentication, though.

When you log in manually, does your browser pop up a little, rather
plain-looking, separate window? Or do you type directly into a form
on the web page itself? If the former, it's probably Basic auth., and
what you're doing should work (or, unlikely, Digest auth., in which
case I think you need urllib2). If the latter, you probably need to
submit an HTML form in the web page to log in.

Some examples on auth and proxies with urllib2 (beware: I don't use a
proxy or basic / digest auth. very often, so these are untested
examples: if you use them, *please* comment on them to say whether
they do or do not work as advertised):

http://www.python.org/sf/798244

To fill in HTML forms, you can use urllib2.urlopen (url,
urllib.urlencod e(...read the docs <wink>...)), or, if you want Python
to parse the form(s) for you and/or don't want to know the messy
details of HTML forms, you could use

http://wwwsearch.sf.net/ClientForm/

You may also find you need to handle HTTP cookies:

http://wwwsearch.sf.net/ClientCookie/

John

[Paul Rubin]

Re: Web authentication

"luigipaior o" <luigipaioro@li bero.it> writes:[color=blue]
> Does anybody know how can I acces to my site with authentication?
>
> I think that urllib2 can help me but I don't undestand how!!![/color]

It's documented in the manual. Try something like (untested):

import urllib

class Open_with_auth( urllib.FancyURL opener):
def prompt_user_pas swd(self, host, realm):
return ('username', 'userpassword') # the uid and passwd you want to use

urllib._urlopen er = Open_with_auth( )

[John J. Lee]

Re: Web authentication

Paul Rubin <http://phr.cx@NOSPAM.i nvalid> writes:
[color=blue]
> "luigipaior o" <luigipaioro@li bero.it> writes:[color=green]
> > Does anybody know how can I acces to my site with authentication?
> >
> > I think that urllib2 can help me but I don't undestand how!!![/color]
>
> It's documented in the manual. Try something like (untested):
>
> import urllib
>
> class Open_with_auth( urllib.FancyURL opener):
> def prompt_user_pas swd(self, host, realm):
> return ('username', 'userpassword') # the uid and passwd you want to use
>
> urllib._urlopen er = Open_with_auth( )[/color]

Doesn't/shouldn't http://user:passwd@example.com/blah.html work?

I don't know where that syntax is specified (if anywhere) -- do you
know, Paul? It seems at a glance that urllib understands that syntax
for ordinary Basic Auth., where urlib2 only knows it as a syntax for
proxy Basic Auth., but I may be wrong there...


John

[Paul Rubin]

Re: Web authentication

jjl@pobox.com (John J. Lee) writes:[color=blue]
> Doesn't/shouldn't http://user:passwd@example.com/blah.html work?[/color]

It ought to but I don't know if urllib supports it. I've always done
it the other way, with that FancyURLOpener subclass.
[color=blue]
> I don't know where that syntax is specified (if anywhere) -- do you
> know, Paul?[/color]

Part of the http spec.

[Alan Kennedy]

Re: Web authentication

[John J. Lee][color=blue]
> Doesn't/shouldn't http://user:passwd@example.com/blah.html work?
>
> I don't know where that syntax is specified (if anywhere)[/color]

RFC 2396: Uniform Resource Identifiers (URI): Generic Syntax

Section: 3.2.2. Server-based Naming Authority

Quoting from that section

"""
URL schemes that involve the direct use of an IP-based protocol to
a
specified server on the Internet use a common syntax for the server
component of the URI's scheme-specific data:

<userinfo>@<hos t>:<port>

where <userinfo> may consist of a user name and, optionally,
scheme-
specific information about how to gain authorization to access the
server. The parts "<userinfo> @" and ":<port>" may be omitted.

server = [ [ userinfo "@" ] hostport ]

The user information, if present, is followed by a commercial
at-sign
"@".

userinfo = *( unreserved | escaped |
";" | ":" | "&" | "=" | "+" | "$" | "," )

Some URL schemes use the format "user:passw ord" in the userinfo
field. This practice is NOT RECOMMENDED, because the passing of
authentication information in clear text (such as URI) has proven
to
be a security risk in almost every case where it has been used.
"""

regards,

--
alan kennedy
------------------------------------------------------
check http headers here: http://xhaus.com/headers
email alan: http://xhaus.com/contact/alan

[John J. Lee]

Re: Web authentication

Alan Kennedy <alanmk@hotmail .com> writes:
[color=blue]
> [John J. Lee][color=green]
> > Doesn't/shouldn't http://user:passwd@example.com/blah.html work?
> >
> > I don't know where that syntax is specified (if anywhere)[/color]
>
> RFC 2396: Uniform Resource Identifiers (URI): Generic Syntax
>
> Section: 3.2.2. Server-based Naming Authority
>
> Quoting from that section
>
> """
> URL schemes that involve the direct use of an IP-based protocol to
> a
> specified server on the Internet use a common syntax for the server
> component of the URI's scheme-specific data:
>
> <userinfo>@<hos t>:<port>[/color]
[...]

Oops, how did I miss that?

Thanks


John

[emiliano]

What about using client form with HTTP authentication ?

Hey guys, i was just googling some information about how to use the ClientForm package with a page which requires HTTP basic authentication and i got here :P ... So here is the problem, lets see if anyone here can help me please solving this issue :)

First i open the protected page using the FancyURLopener which support the HTTP basic authentication and pass this object to the ParseResponse function so it parses the corresponding forms :P

opener = urllib.FancyURL opener({})
URLStream = opener.open ( FormURL )

forms = ParseResponse(U RLStream, backwards_compa t=False)
form = forms[0]

Well, now here i fill in the form ... And finally i try to make the post using the ClientForm objects:

request2 = form.click()
response2 = urllib2.urlopen (request2)

The problem here is that i get the following exception:

"httplib.Invali dURL: nonnumeric port: 'XXXXXXXXXXXX@w ww.domain.com'"

I think that the problem here is that the ClientForm when calling click() tries to build a common urllib Request object which is obvious can't be done cause the provided url to the FancyURLopener which is from were the ClientForm retrieves the url for the Request object has the url username and password :(

So ... Can someone give me some hints or even better some example code to deal with this issue ?

Ps: I don't seem to have a .py file but a .egg file for the ClientForm package so i'm not able to debug or change the ClientForm source :(

/s