httplib continuation packets

[Haakon Riiser]

After a long debugging session while scripting my webmail,
I believe I have traced the problem to the way httplib sends
POST requests.

I have compared tcpdump listings from Python 2.4.3 and 2.5.0's
httplib (via urllib/urllib2), Perl's LWP::UserAgent 2.033 and
Firefox 2.0. Only Python sends the request in such a way that
the mailserver closes the connection before I get any data from
the POST request (immediate FIN packet after the POST request).

httplib always sends the urlencoded POST data in a separate packet
from the HTTP headers, and this seems to cause problems with
the web interface in Ipswitch-IMail/8.05 (the software running on
Doteasy's webmail).

Firefox 2.0 has most of the headers in a single packet, but unlike
httplib, it always places a couple of headers in the continuation
packet as well (usually the content-length and content-type
headers).

LWP::UserAgent 2.033 doesn't use continuation at all, and sends
everything in a single packet.

Is this a bug in httplib or the web server? Is there a
workaround, or should I use Perl for this?

--
Haakon

[Fredrik Lundh]

Re: httplib continuation packets

Haakon Riiser wrote:

Is this a bug in httplib or the web server?

it could be that they're blocking requests from Python's urllib, of
course. have you tried overriding the user-agent string ?

</F>

[Haakon Riiser]

Re: httplib continuation packets

[Fredrik Lundh]

Haakon Riiser wrote:
>

>Is this a bug in httplib or the web server?

>
it could be that they're blocking requests from Python's urllib, of
course. have you tried overriding the user-agent string ?

Yes, and it doesn't help.

By the way, this is the closest thing I've found in the bug tracker:

https://sourceforge.net/tracker/?func=detail&atid=105470&aid=547093&group_id=5470

The bug was closed in 2002 with this comment:

"I changed httplib to send requests as a single packet in rev
1.60. The change was made to address a performance problem,
but happens to fix the problem you had with the bogus
server, too."

Has someone changed it back since then?

--
Haakon

[Fredrik Lundh]

Re: httplib continuation packets

Haakon Riiser wrote:

Yes, and it doesn't help.

then the server is mostly likely broken beyond repair.

By the way, this is the closest thing I've found in the bug tracker:

https://sourceforge.net/tracker/?func=detail&atid=105470&aid=547093&group_id=5470

The bug was closed in 2002 with this comment:
>
"I changed httplib to send requests as a single packet in rev
1.60. The change was made to address a performance problem,
but happens to fix the problem you had with the bogus
server, too."
>
Has someone changed it back since then?

nope; that change buffers the *header* part of the request to avoid
problems with certain TCP/IP mechanisms; see

http://svn.python.org/view?rev=27644&view=rev

for a discussion. note that there's still no guarantee that the entire
header is sent in a single TCP packet.

to see if this really is the problem, you could try moving the call to
self._send_outp ut() from the end of the endheaders() method to the end
of the _send_request() method (around line 870 in httplib.py, at least
in 2.5).

</F>

[Haakon Riiser]

Re: httplib continuation packets

[Fredrik Lundh]

Haakon Riiser wrote:
>

>Yes, and it doesn't help.

>
then the server is mostly likely broken beyond repair.

It's not in my power to upgrade the server, unfortunately.
Guess I'll have to use Perl.

to see if this really is the problem, you could try moving the call to
self._send_outp ut() from the end of the endheaders() method to the end
of the _send_request() method (around line 870 in httplib.py, at least
in 2.5).

Tried this, but the tcpdump still looks the same (two packets: one
with the headers, one with the body), and now it fails with

urllib2.HTTPErr or: HTTP Error 501: Not Implemented

Nevertheless, I'm fairly sure that the packet fragmentation is
the culprit. It works perfectly with Perl, even when I make
no effort at all to spoof the browser (no user-agent, referer,
cookies, etc.).

--
Haakon

[Steve Holden]

Re: httplib continuation packets

Haakon Riiser wrote:

[Fredrik Lundh]
>

>Haakon Riiser wrote:
>>

>>Yes, and it doesn't help.

>then the server is mostly likely broken beyond repair.

>
It's not in my power to upgrade the server, unfortunately.
Guess I'll have to use Perl.
>

>to see if this really is the problem, you could try moving the call to
>self._send_out put() from the end of the endheaders() method to the end
>of the _send_request() method (around line 870 in httplib.py, at least
>in 2.5).

>
Tried this, but the tcpdump still looks the same (two packets: one
with the headers, one with the body), and now it fails with
>
urllib2.HTTPErr or: HTTP Error 501: Not Implemented
>
Nevertheless, I'm fairly sure that the packet fragmentation is
the culprit. It works perfectly with Perl, even when I make
no effort at all to spoof the browser (no user-agent, referer,
cookies, etc.).
>

It really does seem quite bizarre that a server should respond
differently to the same TCP request when it is split differently into IP
datagrams.

There really is nothing wrong (from a standards point of view) with
sending FIN with your last data segment. FIN means "I guarantee to send
no more data, and will continue to acknowledge your segments until I see
your FIN".

Are you planning to report this bug to Ipswitch? It certainly sounds
like someone should.

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://holdenweb.blogspot.com
Recent Ramblings http://del.icio.us/steve.holden

[Haakon Riiser]

Re: httplib continuation packets

[Steve Holden]

It really does seem quite bizarre that a server should respond
differently to the same TCP request when it is split differently into IP
datagrams.
>
There really is nothing wrong (from a standards point of view) with
sending FIN with your last data segment. FIN means "I guarantee to send
no more data, and will continue to acknowledge your segments until I see
your FIN".

It is the server that sends the FIN. What happens is this (each
line corresponds to one packet):

client: POST request headers
client: POST request body
server: FIN + ACK

On receiving the FIN + ACK packet, Python gets immediate
end-of-file on the POST request. Unless the order of the
parameters in the POST request matters (I haven't yet tested this),
I have no other explanation than the fragmentation. If Ipswitch
bothers to reply to my bug report, I'll look into it. Otherwise,
I'm not wasting any more time on this -- it's not that big a deal
for me personally, since I have already scripted the stuff I needed
with Perl.

Are you planning to report this bug to Ipswitch? It certainly sounds
like someone should.

I quickly browsed through ipswitch.com, but couldn't find any good
place to submit bugs. I ended up using the product feedback web
form. Wrote a one-line summary, and referred to this thread on
Google Groups.

--
Haakon