using cgi form via http and extracting results

[bmgx]

I would like to use an already existing online service (currency
converter) basically consisting of a html form with a few options that
is submitted and returns the results. I really don't know where to start
but I have assumed the following steps need to be taken:

1) Make an http connection to the remote script (http://server/script.cgi)

2) Fill out the html form (1x TEXT and 2x SELECT input fields)

3) Submit the form

4) extract the actual returned result. (using regex or something..)

Any help on which modules to use would be much appreciated!

[bmgx]

Re: using cgi form via http and extracting results

I found a possible solution:

http://wwwsearch.sourceforge.net/ClientForm

bmgx wrote:
[color=blue]
> I would like to use an already existing online service (currency
> converter) basically consisting of a html form with a few options that
> is submitted and returns the results. I really don't know where to start
> but I have assumed the following steps need to be taken:
>
> 1) Make an http connection to the remote script (http://server/script.cgi)
>
> 2) Fill out the html form (1x TEXT and 2x SELECT input fields)
>
> 3) Submit the form
>
> 4) extract the actual returned result. (using regex or something..)
>
> Any help on which modules to use would be much appreciated!
>[/color]

[Tim Roberts]

Re: using cgi form via http and extracting results

bmgx <bmgx_no_sp@mgl eesonprop.co.za > wrote:
[color=blue]
>I would like to use an already existing online service (currency
>converter) basically consisting of a html form with a few options that
>is submitted and returns the results. I really don't know where to start
>but I have assumed the following steps need to be taken:
>
>1) Make an http connection to the remote script (http://server/script.cgi)
>
>2) Fill out the html form (1x TEXT and 2x SELECT input fields)
>
>3) Submit the form
>
>4) extract the actual returned result. (using regex or something..)[/color]

You don't actually need steps 1 and 2 at all. HTTP transactions are all
completely separate. The results of a form submission are just a URL. If
the form has fields "FromCurren cy", "ToCurrency ", and "Amount", all you
have to do is this:


http://www.currencyservice.com?FromC...unds&Amount=15

You don't have to HAVE the form source in order to submit a request.
--
- Tim Roberts, timr@probo.com
Providenza & Boekelheide, Inc.

[Ben Finney]

Re: using cgi form via http and extracting results

On Mon, 05 Jan 2004 17:39:30 -0800, Tim Roberts wrote:[color=blue]
> bmgx <bmgx_no_sp@mgl eesonprop.co.za > wrote:[color=green]
>>1) Make an http connection to the remote script (http://server/script.cgi)
>>2) Fill out the html form (1x TEXT and 2x SELECT input fields)
>>3) Submit the form
>>4) extract the actual returned result. (using regex or something..)[/color]
>
> You don't actually need steps 1 and 2 at all.[/color]

True.
[color=blue]
> HTTP transactions are all completely separate.[/color]

True (ignoring cookies for now).
[color=blue]
> The results of a form submission are just a URL.[/color]

Usually false.

The result of most form submissions is an HTTP POST request, which
contains the URL and form content as separate data.
[color=blue]
> If the form has fields "FromCurren cy", "ToCurrency ", and "Amount", all
> you have to do is this:
>
> http://www.currencyservice.com?FromC...unds&Amount=15[/color]

This is only true if the form action is an HTTP GET request, which is a
rather insecure and ugly way to submit form data, and makes it
impossible to submit many kinds of data. HTTP POST is the recommended
method for submitting data to a CGI script.
[color=blue]
> You don't have to HAVE the form source in order to submit a request.[/color]

True. You just need to construct the HTTP POST request correctly.

--
\ "It's a good thing we have gravity or else when birds died |
`\ they'd just stay right up there. Hunters would be all |
_o__) confused." -- Steven Wright |
Ben Finney <http://bignose.squidly .org/>

[bmgx]

Re: using cgi form via http and extracting results

How would one "create" post data so that I can skip the html form and
access the submission script (in <form method="POST" action="SCRIPT" >)
directly. This is simple if the script uses GET as mentioned, but
unfortunately this is not the case which is the very reason for my dilemma..

Ben Finney wrote:[color=blue]
> On Mon, 05 Jan 2004 17:39:30 -0800, Tim Roberts wrote:
>[color=green]
>>bmgx <bmgx_no_sp@mgl eesonprop.co.za > wrote:
>>[color=darkred]
>>>1) Make an http connection to the remote script (http://server/script.cgi)
>>>2) Fill out the html form (1x TEXT and 2x SELECT input fields)
>>>3) Submit the form
>>>4) extract the actual returned result. (using regex or something..)[/color]
>>
>>You don't actually need steps 1 and 2 at all.[/color]
>
>
> True.
>
>[color=green]
>>HTTP transactions are all completely separate.[/color]
>
>
> True (ignoring cookies for now).
>
>[color=green]
>>The results of a form submission are just a URL.[/color]
>
>
> Usually false.
>
> The result of most form submissions is an HTTP POST request, which
> contains the URL and form content as separate data.
>
>[color=green]
>>If the form has fields "FromCurren cy", "ToCurrency ", and "Amount", all
>>you have to do is this:
>>
>>http://www.currencyservice.com?FromC...unds&Amount=15[/color]
>
>
> This is only true if the form action is an HTTP GET request, which is a
> rather insecure and ugly way to submit form data, and makes it
> impossible to submit many kinds of data. HTTP POST is the recommended
> method for submitting data to a CGI script.
>
>[color=green]
>>You don't have to HAVE the form source in order to submit a request.[/color]
>
>
> True. You just need to construct the HTTP POST request correctly.
>[/color]

[bmgx]

Re: using cgi form via http and extracting results

Ok it's done right..

import urllib
a = urllib.urlopen( "http://www.suckerservi ce.com/convert.cgi",
"Amount=1&From= EUR&To=USD")
b = a.readlines()
c = open("tempresul t.html","w")

i = 0
d = "********* local copy ******\n\n"
while i < len(b):
d = d + b[i]
i = i + 1

c.write(d)
c.close()
a.close()

bmgx wrote:
[color=blue]
> How would one "create" post data so that I can skip the html form and
> access the submission script (in <form method="POST" action="SCRIPT" >)
> directly. This is simple if the script uses GET as mentioned, but
> unfortunately this is not the case which is the very reason for my
> dilemma..
>
> Ben Finney wrote:
>[color=green]
>> On Mon, 05 Jan 2004 17:39:30 -0800, Tim Roberts wrote:
>>[color=darkred]
>>> bmgx <bmgx_no_sp@mgl eesonprop.co.za > wrote:
>>>
>>>> 1) Make an http connection to the remote script
>>>> (http://server/script.cgi)
>>>> 2) Fill out the html form (1x TEXT and 2x SELECT input fields)
>>>> 3) Submit the form
>>>> 4) extract the actual returned result. (using regex or something..)
>>>
>>>
>>> You don't actually need steps 1 and 2 at all.[/color]
>>
>>
>>
>> True.
>>
>>[color=darkred]
>>> HTTP transactions are all completely separate.[/color]
>>
>>
>>
>> True (ignoring cookies for now).
>>
>>[color=darkred]
>>> The results of a form submission are just a URL.[/color]
>>
>>
>>
>> Usually false.
>>
>> The result of most form submissions is an HTTP POST request, which
>> contains the URL and form content as separate data.
>>
>>[color=darkred]
>>> If the form has fields "FromCurren cy", "ToCurrency ", and "Amount", all
>>> you have to do is this:
>>>
>>> http://www.currencyservice.com?FromC...unds&Amount=15
>>>[/color]
>>
>>
>>
>> This is only true if the form action is an HTTP GET request, which is a
>> rather insecure and ugly way to submit form data, and makes it
>> impossible to submit many kinds of data. HTTP POST is the recommended
>> method for submitting data to a CGI script.
>>
>>[color=darkred]
>>> You don't have to HAVE the form source in order to submit a request.[/color]
>>
>>
>>
>> True. You just need to construct the HTTP POST request correctly.
>>[/color]
>[/color]

[Joe Francia]

Re: using cgi form via http and extracting results

bmgx wrote:[color=blue]
> Ok it's done right..
>
> import urllib
> a = urllib.urlopen( "http://www.suckerservi ce.com/convert.cgi",
> "Amount=1&From= EUR&To=USD")
> b = a.readlines()
> c = open("tempresul t.html","w")
>
> i = 0
> d = "********* local copy ******\n\n"
> while i < len(b):
> d = d + b[i]
> i = i + 1
>
> c.write(d)
> c.close()
> a.close()
>[/color]

Instead of constructing the data string by hand, you can pass a
dictionary or list of 2-item tuples to urllib.urlencod e() and get a
properly formatted string:

my_data = urllib.urlencod e({'Amount':1,' From':'EUR','To ':'USD'}) # or...
my_data = urllib.urlencod e([('Amount',1),(' From','EUR'),(' To','USD')])
a = urllib.urlopen( 'http://www.suckerservi ce.com/convert.cgi', my_data)

[Tim Roberts]

Re: using cgi form via http and extracting results

Ben Finney <bignose-hates-spam@and-benfinney-does-too.id.au> wrote:
[color=blue]
>On Mon, 05 Jan 2004 17:39:30 -0800, Tim Roberts wrote:[color=green]
>> bmgx <bmgx_no_sp@mgl eesonprop.co.za > wrote:[color=darkred]
>>>1) Make an http connection to the remote script (http://server/script.cgi)
>>>2) Fill out the html form (1x TEXT and 2x SELECT input fields)
>>>3) Submit the form
>>>4) extract the actual returned result. (using regex or something..)[/color]
>>
>> You don't actually need steps 1 and 2 at all.[/color]
>
>True.
>[color=green]
>> HTTP transactions are all completely separate.[/color]
>
>True (ignoring cookies for now).
>[color=green]
>> The results of a form submission are just a URL.[/color]
>
>Usually false.
>
>The result of most form submissions is an HTTP POST request, which
>contains the URL and form content as separate data.[/color]

You are correct. However, in my experience, most handlers that expect POST
data will also accept the same information via GET. That is fortunate,
since it makes it easy to do things the way I suggested.
[color=blue][color=green]
>> If the form has fields "FromCurren cy", "ToCurrency ", and "Amount", all
>> you have to do is this:
>>
>> http://www.currencyservice.com?FromC...unds&Amount=15[/color]
>
>This is only true if the form action is an HTTP GET request, which is a
>rather insecure and ugly way to submit form data, and makes it
>impossible to submit many kinds of data. HTTP POST is the recommended
>method for submitting data to a CGI script.[/color]

There is no such "recommendation ". GET is entirely appropriate for many
kinds of information. A currency conversion service such as the one the
O.P. described is a perfect example. There is no need for security in such
a service, and it makes it easy to do canned requests like the one above.
--
- Tim Roberts, timr@probo.com
Providenza & Boekelheide, Inc.

[Paul Rubin]

Re: using cgi form via http and extracting results

Basically you have to examine the html source for the page you want to
automate, and figure out how the form works. To do that, you need to
understand html and http, there is no way around it. Maybe someone
here can suggest a good reference for that.

Once you've figured out the form, just use the urllib module to build
and send the appropriate http request.