urllib problem (maybe bugs?)

[Timothy Wu]

Hi,

I'm trying to fill the form on page
http://www.cbs.dtu.dk/services/TMHMM/ using urllib.

There are two peculiarities. First of all, I am filling in incorrect
key/value pairs in the parameters on purpose because that's the only
way I can get it to work.. For "version" I am suppose to leave it
unchecked, having value of empty string. And for name "outform" I am
suppose to assign it a value of "-short". Instead, I left out
"outform" all together and fill in "-short" for version. I discovered
the method my accident.

After I've done that it works fine for small SEQ values. Then, when I
try to send large amount of data (1.4MB), it fails miserably with
AttributeError exception.

I highly suspect the two problems I have are the result of some bugs
in the urllib module. Any suggestions?

This is my code:

=============== =============== =============== ===============
fd = open('secretory 0_1.txt', 'r')
txt = fd.read()
fd.close()

params = urllib.urlencod e({'SEQ': txt, 'configfile':
'/usr/opt/www/pub/CBS/services/TMHMM-2.0/TMHMM2.cf', 'version':
'-short'})
f = urllib.urlopen( "http://www.cbs.dtu.dk/cgi-bin/nph-webface", params)
data = f.read()

start = data.find('foll ow <a href="h') + 16
end = data.find('">Th is link</a>')
secondurl = data[start:end]

f = urllib.urlopen( secondurl)
print f.read()
=============== =============== =============== ===============


The value pairs I am suppose to fill in are:

SEQ => some sequence here
configfile => '/usr/opt/www/pub/CBS/services/TMHMM-2.0/TMHMM2.cf'
version => ''
outform => '-short'


The exception I get when sending secretory0_1.tx t is:

C:\Documents and Settings\thw\®à ±>python testhttp.py
Traceback (most recent call last):
File "testhttp.p y", line 11, in ?
f = urllib.urlopen( "http://www.cbs.dtu.dk/cgi-bin/nph-webface", params)
File "C:\Python24\li b\urllib.py", line 79, in urlopen
return opener.open(url , data)
File "C:\Python24\li b\urllib.py", line 182, in open
return getattr(self, name)(url, data)
File "C:\Python24\li b\urllib.py", line 307, in open_http
return self.http_error (url, fp, errcode, errmsg, headers, data)
File "C:\Python24\li b\urllib.py", line 322, in http_error
return self.http_error _default(url, fp, errcode, errmsg, headers)
File "C:\Python24\li b\urllib.py", line 550, in http_error_defa ult
return addinfourl(fp, headers, "http:" + url)
File "C:\Python24\li b\urllib.py", line 836, in __init__
addbase.__init_ _(self, fp)
File "C:\Python24\li b\urllib.py", line 786, in __init__
self.read = self.fp.read
AttributeError: 'NoneType' object has no attribute 'read'



Timothy

[Tim Roberts]

Re: urllib problem (maybe bugs?)

Timothy Wu <2huggie@gmail. com> wrote:[color=blue]
>
>I'm trying to fill the form on page
>http://www.cbs.dtu.dk/services/TMHMM/ using urllib.
>
>There are two peculiarities. First of all, I am filling in incorrect
>key/value pairs in the parameters on purpose because that's the only
>way I can get it to work.. For "version" I am suppose to leave it
>unchecked, having value of empty string. And for name "outform" I am
>suppose to assign it a value of "-short". Instead, I left out
>"outform" all together and fill in "-short" for version. I discovered
>the method my accident.
>
>After I've done that it works fine for small SEQ values. Then, when I
>try to send large amount of data (1.4MB), it fails miserably with
>AttributeErr or exception.
>
>I highly suspect the two problems I have are the result of some bugs
>in the urllib module. Any suggestions?[/color]

The <form> on that page wants multipart/form-data encoding, where each
parameter is embedded in a separate MIME section. That is common with very
large data, and is _required_ when the form includes a file upload. urllib
doesn't do that. It always sends plain old
application/x-www-form-urlencoded data.

I think you're going to have to roll your own sender.
--
- Tim Roberts, timr@probo.com
Providenza & Boekelheide, Inc.