I have a test script below which I use to fetch urls into strings,
either over https or http. When over https, I use m2crypto.urllib and
when over http I use the standard urllib. Whenever, I import sockets
and setdefaulttimeo ut, however, using m2crypto.urllib tends to cause a
http.BadStatusL ine to be raised, even if the timeout is set to be very
large. All of the documents in the test script can be accessed
publicly.
Any ideas? Is there a better/easier way to get https docs in python?
Thanks,
JDH
import urllib, socket
from cStringIO import StringIO
from M2Crypto import Rand, SSL, m2urllib
#comment out this line and the script generally works, but without it
#my zope process, which is using this code, hangs.
socket.setdefau lttimeout(200)
def url_to_string(s ource):
"""
get url as string, for https and http
"""
if source.startswi th('https:'):
sh = StringIO()
url = m2urllib.FancyU RLopener()
url.addheader(' Connection', 'close')
u = url.open(source )
while 1:
data = u.read()
if not data: break
sh.write(data)
return sh.getvalue()
else:
return urllib.urlopen( source).read()
if __name__=='__ma in__':
s1 = url_to_string(' https://crcdocs.bsd.uch icago.edu/crcdocs/Files/informatics.doc ')
s2 = url_to_string(' http://yahoo.com')
s3 = url_to_string(' https://crcdocs.bsd.uch icago.edu/crcdocs/Files/facepage.doc')
print len(s1), len(s2), len(s3)
Comment