Writing web bots in python

[sudharsh@gmail.com]

Hello,
I hav a question..How do I write a webbot that logs onto some website,
fills text into a textbox and submit that form, Sorry I am a novice in
python, apparently I have to use urllib, but I hav a few queries on
this, What happens when there are more input interfaces..doe s urllib
make somekind of a template when it does so..need more info on
this..links and tutorilas would be appreciated..th anx

Sudharshan S

[Fuzzyman]

Re: Writing web bots in python


sudharsh@gmail. com wrote:[color=blue]
> Hello,
> I hav a question..How do I write a webbot that logs onto some website,
> fills text into a textbox and submit that form, Sorry I am a novice in
> python, apparently I have to use urllib, but I hav a few queries on
> this, What happens when there are more input interfaces..doe s urllib
> make somekind of a template when it does so..need more info on
> this..links and tutorilas would be appreciated..th anx
>[/color]

You should use urllib2 rather than urllib. You can see tutorials on
urllib2, and handling cookies if you need to at :

http://www.voidspace.org.uk/python/a.../urllib2.shtml
http://www.voidspace.org.uk/python/a...ookielib.shtml

*However*, this only handles fetching the web pages and posting data
back. You need something that will understand the web 'form' and fill
it in for you.

You probably want Mechanize *or* something like FormEncode. Google for
these.

All the best,

Fuzzyman

[color=blue]
> Sudharshan S[/color]

[sudharsh@gmail.com]

Re: Writing web bots in python

Thanx for the info..i'll let you know when it works

[Paul McGuire]

Re: Writing web bots in python

<sudharsh@gmail .com> wrote in message
news:1142668903 .336769.184570@ u72g2000cwu.goo glegroups.com.. .[color=blue]
> Hello,
> I hav a question..How do I write a webbot that logs onto some website,
> fills text into a textbox and submit that form, Sorry I am a novice in
> python, apparently I have to use urllib, but I hav a few queries on
> this, What happens when there are more input interfaces..doe s urllib
> make somekind of a template when it does so..need more info on
> this..links and tutorilas would be appreciated..th anx
>
> Sudharshan S
>[/color]

C.Titus Brown gave a quick but impressive demo of twill at the Pycon
lightning talks. I think this will do just what you need.

http://www.idyll.org/~t/www-tools/twill/

-- Paul

[Ben C]

Re: Writing web bots in python

On 2006-03-18, sudharsh@gmail. com <sudharsh@gmail .com> wrote:[color=blue]
> Hello,
> I hav a question..How do I write a webbot that logs onto some website,
> fills text into a textbox and submit that form, Sorry I am a novice in
> python, apparently I have to use urllib, but I hav a few queries on
> this, What happens when there are more input interfaces..doe s urllib
> make somekind of a template when it does so..need more info on
> this..links and tutorilas would be appreciated..th anx[/color]

Not strictly Python related, but if you use Firefox, get the "Tamper
Data" extension for it. This lets you see exactly what Firefox is
submitting in the formdata, and what the urls are of the cgi scripts
etc.

I usually invoke a program called curl from python with os.popen, but
this twill someone has suggested looks interesting.