Web-crawling

[John Bradbury]

I am trying to develop a special putpose crawler using htmllib & urllib.
How do you tell the server application that you are a modern browser and can
handle frames?

Thanks,

john Bradbury

[Rene Pijlman]

Re: Web-crawling

John Bradbury:[color=blue]
>I am trying to develop a special putpose crawler using htmllib & urllib.
>How do you tell the server application that you are a modern browser and can
>handle frames?[/color]

I don't know of any "I can handle frames" header and I don't see why the
server would care, but you could mimic the User-agent header sent by a
modern browser.

--
René Pijlman

[John Bradbury]

Re: Web-crawling

I don't know what is causing the problem, but the site I am accessing is
sending out forms for a browser that has a low resolution and does not
support frames. Excuse my ignorance, but where do you set up the User-agent
header you suggested.

Many thanks for your prompt reply.

John Bradbury

"Rene Pijlman" <reply.in.the.n ewsgroup@my.add ress.is.invalid > wrote in
message news:bretnvcng6 9nqpoeug71jon4o bs0moe63f@4ax.c om...[color=blue]
> John Bradbury:[color=green]
> >I am trying to develop a special putpose crawler using htmllib & urllib.
> >How do you tell the server application that you are a modern browser and[/color][/color]
can[color=blue][color=green]
> >handle frames?[/color]
>
> I don't know of any "I can handle frames" header and I don't see why the
> server would care, but you could mimic the User-agent header sent by a
> modern browser.
>
> --
> René Pijlman[/color]

[Rene Pijlman]

Re: Web-crawling

John Bradbury:[color=blue]
>where do you set up the User-agent header you suggested.[/color]

Its an HTTP header:

http://www.ietf.org/rfc/rfc2616.txt

See also:

http://www.google.com/search?q=urllib+python+%22user%2Dagent%22

which leads you to... tada... the documentation! :-)

http://www.python.org/doc/current/lib/module-urllib.html

--
René Pijlman

[John J. Lee]

Re: Web-crawling

"John Bradbury" <john_bradbury@ ___cableinet.co .uk> writes:
[color=blue]
> "Rene Pijlman" <reply.in.the.n ewsgroup@my.add ress.is.invalid > wrote in
> message news:bretnvcng6 9nqpoeug71jon4o bs0moe63f@4ax.c om...[color=green]
> > John Bradbury:[color=darkred]
> > >I am trying to develop a special putpose crawler using htmllib & urllib.
> > >How do you tell the server application that you are a modern browser
> > >and can handle frames?[/color][/color][/color]
[...][color=blue][color=green]
> > server would care, but you could mimic the User-agent header sent by a[/color][/color]
[...][color=blue]
> I don't know what is causing the problem, but the site I am accessing is
> sending out forms for a browser that has a low resolution and does not
> support frames. Excuse my ignorance, but where do you set up the
> User-agent header you suggested.[/color]

For urllib2 (well, almost):

http://wwwsearch.sourceforge.net/ClientCookie/doc.html#headers

John