Select weirdness

[Ron Garret]

Here's my code. It's a teeny weeny little HTTP server. (I'm not really
trying to reinvent the wheel here. What I'm really doing is writing a
dispatching proxy server, but this is the shortest way to illustrate the
problem I'm having):

from SocketServer import *
from socket import *
from select import select

class myHandler(Strea mRequestHandler ):
def handle(self):
print '>>>>>>>>>>>'
while 1:
sl = select([self.rfile],[],[])[0]
if sl:
l = self.rfile.read line()
if len(l)<3: break
print l,
pass
pass
print>>self.wfi le, 'HTTP/1.0 200 OK'
print>>self.wfi le, 'Content-type: text/plain'
print>>self.wfi le
print>>self.wfi le, 'foo'
self.rfile.clos e()
self.wfile.clos e()
print '<<<<<<<<<<<<'
pass
pass

def main():
server = TCPServer(('',8 080), myHandler)
server.serve_fo rever()
pass

if __name__ == '__main__': main()


If I telnet into this server and type in an HTTP request manually it
works fine. But when I try to access this with a browser (I've tried
Firefox and Safari -- both do the same thing) it hangs immediately after
reading the first line in the request (i.e. before reading the first
header).

When I click the "stop" button in the browser it breaks the logjam and
the server reads the headers (but then of course it dies trying to write
the response to a now-closed socket).

The only difference I can discern is that the browser send \r\n for
end-of-line while telnet just sends \n. But I don't see why that should
make any difference. So I'm stumped. Any clues would be much
appreciated.

Thanks,
rg

[Bjoern Schliessmann]

Re: Select weirdness

Ron Garret wrote:

print>>self.wfi le, 'HTTP/1.0 200 OK'
print>>self.wfi le, 'Content-type: text/plain'
print>>self.wfi le
print>>self.wfi le, 'foo'
[...]
If I telnet into this server and type in an HTTP request manually
it works fine. But when I try to access this with a browser (I've
tried Firefox and Safari -- both do the same thing) it hangs
immediately after reading the first line in the request (i.e.
before reading the first header).

Could the problem be that print ends lines with "\r\n" not on all
platforms? Try doing it explicitly.

The only difference I can discern is that the browser send \r\n
for end-of-line while telnet just sends \n.

Then, I think, your telnet client is broken. I've seem multiple
applications relying on telnet or similar protocols which
required "\r\n" as end-of-line. E. g. I needed to hack
twisted.protoco ls.basic.LineRe ceiverFactory since it showed the
exact behaviour you described for browsers. It showed up because
busybox' telnet only sends \n as EOL.

But I don't see why that should make any difference.

Easy. If you only accept "\r\n" as EOL, you'll wait forever unless
you really receive it.

Regards,


Björn

--
BOFH excuse #81:

Please excuse me, I have to circuit an AC line through my head to
get this database working.

[Irmen de Jong]

Re: Select weirdness

Ron Garret wrote:

Here's my code. It's a teeny weeny little HTTP server. (I'm not really
trying to reinvent the wheel here. What I'm really doing is writing a
dispatching proxy server, but this is the shortest way to illustrate the
problem I'm having):
>
from SocketServer import *
from socket import *
from select import select
>
class myHandler(Strea mRequestHandler ):
def handle(self):
print '>>>>>>>>>>>'
while 1:
sl = select([self.rfile],[],[])[0]
if sl:
l = self.rfile.read line()
if len(l)<3: break
print l,
pass
pass
print>>self.wfi le, 'HTTP/1.0 200 OK'
print>>self.wfi le, 'Content-type: text/plain'
print>>self.wfi le
print>>self.wfi le, 'foo'
self.rfile.clos e()
self.wfile.clos e()
print '<<<<<<<<<<<<'
pass
pass

You shouldn't use a select() of your own inside the handle method.
The socketserver takes care of that for you.

What you need to do is read the input from the socket until you've reached
the end-of-input marker. That marker is an empty line containing just "\r\n"
when dealing with HTTP GET requests.

Any reason don't want to use the BasicHTTPServer that comes with Python?

Anyway, try the following instead:

from SocketServer import *
from socket import *
from select import select

class myHandler(Strea mRequestHandler ):
def handle(self):
print '>>>>>>>>>>>'
while True:
l = self.rfile.read line()
print repr(l)
if not l or l=='\r\n':
break
print>>self.wfi le, 'HTTP/1.0 200 OK'
print>>self.wfi le, 'Content-type: text/plain'
print>>self.wfi le
print>>self.wfi le, 'foo'
self.rfile.clos e()
self.wfile.clos e()
print '<<<<<<<<<<<<'

def main():
server = TCPServer(('',8 080), myHandler)
server.serve_fo rever()

if __name__ == '__main__': main()



--Irmen

[Ron Garret]

Re: Select weirdness

In article <462b3e47$0$326 $e4fe514c@news. xs4all.nl>,
Irmen de Jong <irmen.NOSPAM@x s4all.nlwrote:

Ron Garret wrote:

Here's my code. It's a teeny weeny little HTTP server. (I'm not really
trying to reinvent the wheel here. What I'm really doing is writing a
dispatching proxy server, but this is the shortest way to illustrate the
problem I'm having):

from SocketServer import *
from socket import *
from select import select

class myHandler(Strea mRequestHandler ):
def handle(self):
print '>>>>>>>>>>>'
while 1:
sl = select([self.rfile],[],[])[0]
if sl:
l = self.rfile.read line()
if len(l)<3: break
print l,
pass
pass
print>>self.wfi le, 'HTTP/1.0 200 OK'
print>>self.wfi le, 'Content-type: text/plain'
print>>self.wfi le
print>>self.wfi le, 'foo'
self.rfile.clos e()
self.wfile.clos e()
print '<<<<<<<<<<<<'
pass
pass

>
>
You shouldn't use a select() of your own inside the handle method.
The socketserver takes care of that for you.

I don't understand why socketserver calling select should matter. (And
BTW, there are no calls to select in SocketServer.py . I'm using
Python2.5.)

What you need to do is read the input from the socket until you've reached
the end-of-input marker. That marker is an empty line containing just "\r\n"
when dealing with HTTP GET requests.
>
Any reason don't want to use the BasicHTTPServer that comes with Python?

Yes, but it's a long story. What I'm really doing is writing a
dispatching proxy. It reads the HTTP request and then redirects it to a
number of different servers depending on the URL. The servers are all
local and served off of unix sockets, not TCP sockets (which is why I
can't just configure Apache to do this).

The reason I'm running this weird configuration (in case you're
wondering) is because this is a development machine and multiple copies
of the same website run on it simultaneously. Each copy of the site
runs a customized server to handle AJAX requests efficiently.

>
Anyway, try the following instead:
>

That won't work for POST requests.

rg

[Ron Garret]

Re: Select weirdness

In article <590s1nF2ibkeqU 1@mid.individua l.net>,
Bjoern Schliessmann <usenet-mail-0306.20.chr0n0s s@spamgourmet.c om>
wrote:

The only difference I can discern is that the browser send \r\n
for end-of-line while telnet just sends \n.

....

But I don't see why that should make any difference.

>
Easy. If you only accept "\r\n" as EOL, you'll wait forever unless
you really receive it.

But it's SELECT that is hanging. Select knows (or ought to know)
nothing of end-of-line markers. Readline (when it is called) is doing
the Right Thing, and my end-of-request test is (currently) reading a
line less than 3 characters long. All that is working. The problem is
that select is saying there is no input (and therefore hanging) when in
fact there is input.

rg

[Ron Garret]

Re: Select weirdness

In article <rNOSPAMon-36DDEF.23212021 042007@news.gha .chartermi.net> ,
Ron Garret <rNOSPAMon@flow net.comwrote:

Here's my code. It's a teeny weeny little HTTP server. (I'm not really
trying to reinvent the wheel here. What I'm really doing is writing a
dispatching proxy server, but this is the shortest way to illustrate the
problem I'm having):
>
from SocketServer import *
from socket import *
from select import select
>
class myHandler(Strea mRequestHandler ):
def handle(self):
print '>>>>>>>>>>>'
while 1:
sl = select([self.rfile],[],[])[0]
if sl:
l = self.rfile.read line()
if len(l)<3: break
print l,
pass
pass
print>>self.wfi le, 'HTTP/1.0 200 OK'
print>>self.wfi le, 'Content-type: text/plain'
print>>self.wfi le
print>>self.wfi le, 'foo'
self.rfile.clos e()
self.wfile.clos e()
print '<<<<<<<<<<<<'
pass
pass
>
def main():
server = TCPServer(('',8 080), myHandler)
server.serve_fo rever()
pass
>
if __name__ == '__main__': main()
>
>
If I telnet into this server and type in an HTTP request manually it
works fine. But when I try to access this with a browser (I've tried
Firefox and Safari -- both do the same thing) it hangs immediately after
reading the first line in the request (i.e. before reading the first
header).
>
When I click the "stop" button in the browser it breaks the logjam and
the server reads the headers (but then of course it dies trying to write
the response to a now-closed socket).
>
The only difference I can discern is that the browser send \r\n for
end-of-line while telnet just sends \n. But I don't see why that should
make any difference. So I'm stumped. Any clues would be much
appreciated.

I have reproduced the problem using Telnet, so that proves it's not an
EOL issue. The problem seems to be timing-related. If I type the
request in manually it works. If I paste it in all at once, it hangs.
It's actually even weirder than that: if I pipe the request into a
telnet client then it hangs after the request line, just with a browser
(or wget). But if I literally paste the request into a window running a
telnet client, it gets past the request line and four out of seven
headers before it hangs.

rg

[Ron Garret]

Re: Select weirdness

I think I've figured out what's going on.

First, here's the smoking gun: I changed the code as follows:

class myHandler(Strea mRequestHandler ):
def handle(self):
print '>>>>>>>>>>>'
while 1:
sl = select([self.rfile],[],[],1)[0]
print sl
l = self.rfile.read line()
if len(l)<3: break
print l,
pass

(Rest of code is unchanged.)

In other words, I select on the rfile object and added a timeout.

Here's the result when I cut-and-paste an HTTP request:

>>>>>>>>>>>

[<socket._fileob ject object at 0x6b110>]
GET /foo/baz HTTP/1.1
[<socket._fileob ject object at 0x6b110>]
Accept: */*
[<socket._fileob ject object at 0x6b110>]
Accept-Language: en
[<socket._fileob ject object at 0x6b110>]
Accept-Encoding: gzip, deflate
[<socket._fileob ject object at 0x6b110>]
Cookie: c1=18:19:55.042 196; c2=18:19:55.042 508
[]
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en)
AppleWebKit/419 (KHTM
[]
L, like Gecko) Safari/419.3
[]
Connection: keep-alive
[]
Host: localhost:8081
[]
<<<<<<<<<<<<

As you can see, the select call shows input available for a while (five
lines) and then shows no input available despite the fact that there is
manifestly still input available.

The answer is obvious: select is looking only at the underlying socket,
and not at the rfile buffers.

So... is this a bug in select? Or a bug in my code?

rg

[Ron Garret]

Bug in select (was: Re: Select weirdness)

In article <rNOSPAMon-934744.11420922 042007@news.gha .chartermi.net> ,
Ron Garret <rNOSPAMon@flow net.comwrote:

The answer is obvious: select is looking only at the underlying socket,
and not at the rfile buffers.

Here is conclusive proof that there's a bug in select:

from socket import *
from select import select
s=socket(AF_INE T, SOCK_STREAM)
s.bind(('',8080 ))
s.listen(5)
f = s.accept()[0].makefile()
# Now telnet to port 8080 and enter two lines of random text
f.readline()
select([f],[],[],1)
f.readline()

Here's the sample input:

[ron@mickey:~/devel/sockets]$ telnet localhost 8081
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
123
321


And this is the result:

>>f.readline( )

'123\r\n'

>>select([f],[],[],1)

# After one second...
([], [], [])

>>f.readline( )

'321\r\n'

>>>

So this is clearly a bug, but surely I'm not the first person to have
encountered this? Is there a known workaround?

rg

[Erik Max Francis]

Re: Bug in select

Ron Garret wrote:

So this is clearly a bug, but surely I'm not the first person to have
encountered this? Is there a known workaround?

It's hard to see how this demonstrates a bug in anything, since you're
telnetting to the wrong port in your example.

--
Erik Max Francis && max@alcyone.com && http://www.alcyone.com/max/
San Jose, CA, USA && 37 20 N 121 53 W && AIM, Y!M erikmaxfrancis
Being in love for real / It ain't like a movie screen
-- India Arie

[Ron Garret]

Re: Bug in select

In article <1aGdndBf1ON7Ur bbnZ2dnUVZ_hjin Z2d@speakeasy.n et>,
Erik Max Francis <max@alcyone.co mwrote:

Ron Garret wrote:
>

So this is clearly a bug, but surely I'm not the first person to have
encountered this? Is there a known workaround?

>
It's hard to see how this demonstrates a bug in anything, since you're
telnetting to the wrong port in your example.

Geez you people are picky. Since I ran this several times I ran into
the TIM_WAIT problem. Here's the actual transcript:

Python 2.5 (r25:51908, Mar 1 2007, 10:09:05)
[GCC 4.0.1 (Apple Computer, Inc. build 5367)] on darwin
Type "help", "copyright" , "credits" or "license" for more information.

>>from socket import *
>>from select import select
>>s=socket(AF_I NET, SOCK_STREAM)
>>s.bind(('',80 80))

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 1, in bind
socket.error: (48, 'Address already in use')

>>s.bind(('',80 81))
>>s.listen(5)
>>f = s.accept()[0].makefile()
>>f.readline( )

'123\r\n'

>>select([f],[],[],1)

([], [], [])

>>f.readline( )

'321\r\n'

[Erik Max Francis]

Re: Bug in select

Ron Garret wrote:

Geez you people are picky. Since I ran this several times I ran into
the TIM_WAIT problem. Here's the actual transcript:

It's not about being picky, it's about making it clear what your problem
is. You're now describing an entirely different problem, hence why it's
important to be clear about _precisely_ what it is you're doing and
_precisely_ what's happening that you think that's wrong.

Python 2.5 (r25:51908, Mar 1 2007, 10:09:05)
[GCC 4.0.1 (Apple Computer, Inc. build 5367)] on darwin
Type "help", "copyright" , "credits" or "license" for more information.

>>>from socket import *
>>>from select import select
>>>s=socket(AF_ INET, SOCK_STREAM)
>>>s.bind(('',8 080))

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 1, in bind
socket.error: (48, 'Address already in use')

>>>s.bind(('',8 081))
>>>s.listen(5 )
>>>f = s.accept()[0].makefile()
>>>f.readline ()

'123\r\n'

>>>select([f],[],[],1)

([], [], [])

>>>f.readline ()

'321\r\n'

--
Erik Max Francis && max@alcyone.com && http://www.alcyone.com/max/
San Jose, CA, USA && 37 20 N 121 53 W && AIM, Y!M erikmaxfrancis
There are no dull subjects. There are only dull writers.
-- H.L. Mencken

[Ron Garret]

Re: Bug in select

In article <AaqdnfQ6jfc0fb bbnZ2dnUVZ_u3in Z2d@speakeasy.n et>,
Erik Max Francis <max@alcyone.co mwrote:

Ron Garret wrote:
>

Geez you people are picky. Since I ran this several times I ran into
the TIM_WAIT problem. Here's the actual transcript:

>
It's not about being picky, it's about making it clear what your problem
is. You're now describing an entirely different problem,

Nope, it's been the same problem all along. If you don't think so then
you have misunderstood something.

rg

[Ron Garret]

Re: Bug in select (was: Re: Select weirdness)

In article <3dRWh.4178$j63 .322@newsread2. news.pas.earthl ink.net>,
Dennis Lee Bieber <wlfraed@ix.net com.comwrote:

Well, on WinXP, Python 2.4, with

I should have specified: I'm running 2.5 on unix. (I've reproduced the
problem on both Linux and OS X.)

rg

[Irmen de Jong]

Re: Select weirdness

Ron Garret wrote:

I don't understand why socketserver calling select should matter. (And
BTW, there are no calls to select in SocketServer.py . I'm using
Python2.5.)

You don't *need* a select at all.
Socketserver just blocks on accept() and dispatches a handler
on the new connection.

>Anyway, try the following instead:
>>

>
That won't work for POST requests.
>

Why not?
Just add some more code to deal with the POST request body.
There should be a content-length header to tell you how many
bytes to read after the header section has finished.

--Irmen

[Ron Garret]

Re: Select weirdness

In article <462c54cb$0$336 $e4fe514c@news. xs4all.nl>,
Irmen de Jong <irmen.NOSPAM@x s4all.nlwrote:

Ron Garret wrote:

I don't understand why socketserver calling select should matter. (And
BTW, there are no calls to select in SocketServer.py . I'm using
Python2.5.)

>
You don't *need* a select at all.

Yes I do, because what I'm really writing is a dispatching proxy that
has to serve many simultaneous connections.

Here's the full story in case you're interested: We have an application
that is currently fielded as a cgi. We have a development server that
needs to run multiple copies of the application at the same time. This
is so that developers can push changes into their private "sandboxes"
for evaluation before going into the main development branch. Each
sandbox has its own source tree, its own database, and its own URL
namespace.

There are a small number of URLs in the application that are performance
bottlenecks (they are used to serve AJAX updates). In order to
alleviate that bottleneck without having to rewrite the whole
application to run under mod_python or some such thing we've written a
special dedicated server that handles only the AJAX requests.

The tricky part is that each developer needs to have their own copy of
this server running because each developer can have different code that
needs to run to serve those requests. Assigning each developer a
dedicated IP port would be a configuration nightmare, so these servers
serve run on unix sockets rather than TCP sockets.

I have not been able to find a proxy server that can proxy to unix
sockets, so I need to write my own. Conceptually its a very simple
thing: read the first line of an HTTP request, parse it with a regexp to
extract the sandbox name, connect to the appropriate unix server socket,
and then bidirectionally pipe bytes back and forth. But it has to do
this for multiple connections simultaneously, which is why I need select.

Anyway, try the following instead:
>

That won't work for POST requests.

>
Why not?

Because POST requests can be very complicated.

Just add some more code to deal with the POST request body.

I was really hoping to avoid having to write a fully HTTP-aware proxy.

There should be a content-length header to tell you how many
bytes to read after the header section has finished.

Not if the content-transfer-encoding is chunked. Or if there are
multiple file attachments.

Also, even GET requests can become very complicated (from a protocol
point of view) in HTTP 1.1.

rg