Building browser-like GET request

**=?iso-8859-1?B?Qmr2cm4gS2VpbA==?=** · Apr 21 '07, 09:55 PM

Re: Building browser-like GET request

On 21 Apr., 23:28, Gilles Ganault <nos...@nospam. comwrote:

I'd like to download pages from a site, but it checks whether
the requests are coming from a live user or a script; If the latter,
the server returns a blank page.
>
Using a proxy (Paros), I can see what information my script and
FireFox send, and there are a lot of information that Python is
missing:

Well, I am brand new to Python, so it takes me a lot of guessing, but
since it seems you're using urlib2:

On http://docs.python.org/lib/module-urllib2.html is written that you
may add custom headers to your http requests.
Either by calling "addheader( )" or by passing a dictionary with
headers to the constructor.

I hope that helped and I wasn't telling things you already new.
As a sidenote: For the task you describe I'd rather use an actual
sniffer - such as Wireshark (http://en.wikipedia.org/wiki/Wireshark),
than logs of a Proxy... Not sure wether Wireshark works under Windows,
though.

Good luck!

**Gilles Ganault** · Apr 21 '07, 10:15 PM

Re: Building browser-like GET request

On 21 Apr 2007 14:47:55 -0700, Björn Keil <abgrund@silber drache.net>
wrote:

>Well, I am brand new to Python, so it takes me a lot of guessing, but
>since it seems you're using urlib2:

Thanks. Indeed, it looks like urlib2 is the way to go when going
through a proxy.

For those interested, here's how to download a page through a proxy:

----------------------------
import sys
import urllib
import urllib2
import re

#set up proxy
proxy_info = { 'host' : 'localhost','po rt' : 8080}
proxy_support = urllib2.ProxyHa ndler({"http" :
"http://%(host)s:%(port )d" % proxy_info})
opener = urllib2.build_o pener(proxy_sup port)
urllib2.install _opener(opener)

#call page with specific headers
url = 'http://www.acme.com/cgi-bin/read?code=123'
headers = {
'User-Agent' : 'Mozilla/4.0 (compatible; MSIE 5.5; Windows
NT)',
'Accept' :
'text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,ima ge/png,*/*;q=0.5',
'Accept-Language' : 'fr-fr,en-us;q=0.7,en;q=0 .3',
'Accept-Charset' : 'ISO-8859-1,utf-8;q=0.7,*;q=0.7 '
}
#None = GET; set values to use POST
req = urllib2.Request (url, None, headers)

response = urllib2.urlopen (req).read()
log = open('output.ht ml','w')
log.write(respo nse)
log.close()
----------------------------

Thanks.

**Steve Holden** · Apr 22 '07, 10:15 PM

Re: Building browser-like GET request

Björn Keil wrote:
[...]

>
I hope that helped and I wasn't telling things you already new.
As a sidenote: For the task you describe I'd rather use an actual
sniffer - such as Wireshark (http://en.wikipedia.org/wiki/Wireshark),
than logs of a Proxy... Not sure wether Wireshark works under Windows,
though.

On a point of information, Wireshark wokrs very effectively under
Windows. The only thing you shouldn't expect to be able to do is tap
into the loopback network, and that's down to the Windows driver structure.

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
Recent Ramblings http://holdenweb.blogspot.com

**Gilles Ganault** · Apr 24 '07, 10:15 PM

Re: Building browser-like GET request

On Sun, 22 Apr 2007 18:07:57 -0400, Steve Holden <steve@holdenwe b.com>
wrote:

>On a point of information, Wireshark wokrs very effectively under
>Windows. The only thing you shouldn't expect to be able to do is tap
>into the loopback network, and that's down to the Windows driver structure.

Thanks for the tip. Someone mentionned a lighter alternative to
display what goes on between browser and web server:

PocketSoap's TCPTrace

@superfell

http://www.pocketsoap.com/tcptrace/

Building browser-like GET request

Building browser-like GET request

Comment

Comment

Comment

Comment