closing a server socket

**Peter Hansen** · Jul 18 '05, 03:42 AM

Re: closing a server socket

simon place wrote:[color=blue]
>
> Spent some very frustrating hours recoding to find a way of closing a server
> socket, i'd not thought it would be any problem,
>
> however, after complete failure and as a last resort, i looked at the python
> wrapper module for sockets, and found that the close command doesn't actually
> call the underlying close! this didn't seem right, so i added it, and my code
> now works simply and as expected.
>
> def close(self):
> self._sock.clos e() # added 2003-oct-13
> self._sock = _closedsocket()
> self.send = self.recv = self.sendto = self.recvfrom = self._sock._dum my
> close.__doc__ = _realsocket.clo se.__doc__
>
> Probably only on win32, the comments in the socket module seem to indicate
> different codings on different platforms.[/color]

None of this should be necessary!

What problems were you having that led you to try the above abomination in
your effort to resolve them?

-Peter

**simon place** · Jul 18 '05, 03:44 AM

Re: closing a server socket

Peter Hansen wrote:[color=blue]
> simon place wrote:
>[color=green]
>>Spent some very frustrating hours recoding to find a way of closing a server
>>socket, i'd not thought it would be any problem,
>>
>>however, after complete failure and as a last resort, i looked at the python
>>wrapper module for sockets, and found that the close command doesn't actually
>>call the underlying close! this didn't seem right, so i added it, and my code
>>now works simply and as expected.
>>
>> def close(self):
>> self._sock.clos e() # added 2003-oct-13
>> self._sock = _closedsocket()
>> self.send = self.recv = self.sendto = self.recvfrom = self._sock._dum my
>> close.__doc__ = _realsocket.clo se.__doc__
>>
>>Probably only on win32, the comments in the socket module seem to indicate
>>different codings on different platforms.[/color]
>
>
> None of this should be necessary!
>
> What problems were you having that led you to try the above abomination in
> your effort to resolve them?
>
> -Peter[/color]

The problem is i couldn't find anyway, in 'normal' code, to close a server
socket, ie unblock an accept.

tried to use non-blocking mode, but not allowed on a server socket, which
makes sense, but isn't documented.

tried to use time-outs, which was close, but seems to close the accepted
connection sockets on a server socket time-out.

tried to fire a dummy connection to unblock the accept, which works, but if
your ip changes, i.e. a dialup, you can't do that.

i only hacked the module because of frustration, but it took about a minute to
spot, and now every thing works the original way i tried, and i've been doing
some thrashing and had no problems.

**Peter Hansen** · Jul 18 '05, 03:44 AM

Re: closing a server socket

simon place wrote:[color=blue]
>
> Peter Hansen wrote:[color=green]
> > None of this should be necessary!
> >
> > What problems were you having that led you to try the above abomination in
> > your effort to resolve them?[/color]
>
> The problem is i couldn't find anyway, in 'normal' code, to close a server
> socket, ie unblock an accept.
>
> tried to use non-blocking mode, but not allowed on a server socket, which
> makes sense, but isn't documented.[/color]

It does work, which is why it isn't documented that it doesn't. <wink>

What did you try, that non-blocking server sockets don't work for you?

(I assume you are talking about using select.select() to detect when the
listening socket has a client connection waiting to be accepted.)

-Peter

**simon place** · Jul 18 '05, 03:57 AM

Re: closing a server socket

i'm trying to see if anyone else thinks this is a bug,

when you call socket.socket.c lose() the wrapper only marks the socket as
closed, so no exception to unblock socket.socket.a ccept(), and you can
actually still get connections on this socket?

( this may only be for windows. )

simon.

**Peter Hansen** · Jul 18 '05, 03:57 AM

Re: closing a server socket

simon place wrote:[color=blue]
>
> i'm trying to see if anyone else thinks this is a bug,
>
> when you call socket.socket.c lose() the wrapper only marks the socket as
> closed, so no exception to unblock socket.socket.a ccept(), and you can
> actually still get connections on this socket?
>
> ( this may only be for windows. )[/color]

A repost? What didn't you like about my previous replies?

I guess you are basically saying that you want to use a blocking
socket (because you can't get non-blocking ones to work, though they
do work in general?) but you still want to be able to terminate
your call from another thread by forcibly closing the socket...

If that's so, give it up. You can't do that because the
accept() call is stuck down in C code which can't be
interrupted from another Python thread. Your only hope would be
to have the server call be in a separate process, which you
could "kill", but then you'd of course have to talk to this
other process with sockets, and be back where you started.

Use non-blocking sockets. They do work.

-Peter

**Alex Martelli** · Jul 18 '05, 03:58 AM

Re: closing a server socket

Peter Hansen wrote:
...[color=blue]
> I guess you are basically saying that you want to use a blocking
> socket (because you can't get non-blocking ones to work, though they
> do work in general?) but you still want to be able to terminate
> your call from another thread by forcibly closing the socket...
>
> If that's so, give it up. You can't do that because the[/color]

VERY good point -- and since by a weird coincidence I've had to deal
with EXACTLY this problem TWICE over the last few days for two
different clients (!), both times in the context of xml-rpc serving,
let me emphasize this once again: that's not the way to do it.

[color=blue]
> Use non-blocking sockets. They do work.[/color]

Or architect a solution where the blocked thread will get out
from its stuck state in the blocking socket "naturally" . That's
quite easy in XML-RPC, e.g. as follows...:

_baseclass = SimpleXMLRPCSer ver.SimpleXMLRP CServer
class TerminatableSer ver(_baseclass) :
allow_reuse_add ress = True

def __init__(self, addr, *args, **kwds):
self.myhost, self.myport = addr
_baseclass.__in it__(self, addr, *args, **kwds)
self.register_f unction(self.te rminate_server)

quit = False
def serve_forever(s elf):
while not self.quit:
self.handle_req uest()
self.server_clo se()

def terminate_serve r(self, authentication_ token):
if not check_as_valid( authentication_ token):
return 1, "Invalid or expired authentication token"
self.quit = True
return 0, "Server terminated on host %r, port %r" % (
self.myhost, self.myport)

(you'll want some way to verify authentication, of course, but
that's probably something you already have somewhere in such an
architecture, so I'm simplistically representing it by the
authentication_ token parameter and a hypothetical check_as_valid
function that's able to check it).

The point is that in this way server termination is architected
as just one transaction -- the self.handle_req uest() in the body
of the while will eventually hand off control to terminate_serve r,
which may set self.quit, and that in turn will terminate the
while loop, since as soon as handle_request is done the loop's
condition is checked again. There are other ways, of course, but
I've found this one clean and reliable for these specific needs.

Alex

**simon place** · Jul 18 '05, 03:59 AM

Re: closing a server socket

Alex Martelli wrote:
[color=blue]
> Peter Hansen wrote:
> ...
>[color=green]
>>I guess you are basically saying that you want to use a blocking
>>socket (because you can't get non-blocking ones to work, though they
>>do work in general?) but you still want to be able to terminate
>>your call from another thread by forcibly closing the socket...
>>
>>If that's so, give it up. You can't do that because the[/color]
>
>
> VERY good point -- and since by a weird coincidence I've had to deal
> with EXACTLY this problem TWICE over the last few days for two
> different clients (!), both times in the context of xml-rpc serving,
> let me emphasize this once again: that's not the way to do it.
>
>
>[color=green]
>>Use non-blocking sockets. They do work.[/color]
>
>
> Or architect a solution where the blocked thread will get out
> from its stuck state in the blocking socket "naturally" . That's
> quite easy in XML-RPC, e.g. as follows...:
>
>
> _baseclass = SimpleXMLRPCSer ver.SimpleXMLRP CServer
> class TerminatableSer ver(_baseclass) :
> allow_reuse_add ress = True
>
> def __init__(self, addr, *args, **kwds):
> self.myhost, self.myport = addr
> _baseclass.__in it__(self, addr, *args, **kwds)
> self.register_f unction(self.te rminate_server)
>
> quit = False
> def serve_forever(s elf):
> while not self.quit:
> self.handle_req uest()
> self.server_clo se()
>
> def terminate_serve r(self, authentication_ token):
> if not check_as_valid( authentication_ token):
> return 1, "Invalid or expired authentication token"
> self.quit = True
> return 0, "Server terminated on host %r, port %r" % (
> self.myhost, self.myport)
>
> (you'll want some way to verify authentication, of course, but
> that's probably something you already have somewhere in such an
> architecture, so I'm simplistically representing it by the
> authentication_ token parameter and a hypothetical check_as_valid
> function that's able to check it).
>
> The point is that in this way server termination is architected
> as just one transaction -- the self.handle_req uest() in the body
> of the while will eventually hand off control to terminate_serve r,
> which may set self.quit, and that in turn will terminate the
> while loop, since as soon as handle_request is done the loop's
> condition is checked again. There are other ways, of course, but
> I've found this one clean and reliable for these specific needs.
>
>
> Alex
>[/color]

Yes, no problem closing a server before it returns to accept another
connection, but when the handler is threaded,

see below, current situation

import socket[color=blue][color=green][color=darkred]
>>> s=socket.socket (socket.AF_INET ,socket.SOCK_ST REAM)
>>> s.bind(('127.0. 0.1',80))
>>> s.listen(1)
>>> import threading
>>> t=threading.Thr ead(target=s.ac cept)
>>> t.start()
>>> s.close()
>>> import inspect
>>> inspect.getsour ce(socket.socke t.close)[/color][/color][/color]
' def close(self):\n self._sock = _closedsocket() \n self.send
= self.recv = self.sendto = self.recvfrom = self._sock._dum my\n'[color=blue][color=green][color=darkred]
>>> t.isAlive()[/color][/color][/color]
True[color=blue][color=green][color=darkred]
>>> s=socket.socket (socket.AF_INET ,socket.SOCK_ST REAM)
>>> s.connect(('127 .0.0.1',80))
>>> t.isAlive()[/color][/color][/color]
False[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

i.e. closing server socket does not unblock accept and you can still get
connections on it!
( because it has not actually been closed!)
how is this acceptable?

with socket modified to actually close the socket,
[color=blue][color=green][color=darkred]
>>> import socket
>>> import inspect
>>> inspect.getsour ce(socket.socke t.close)[/color][/color][/color]
' def close(self):\n self._sock.clos e()\n self._sock = _closeds
ocket()\n self.send = self.recv = self.sendto = self.recvfrom = self._soc
k._dummy\n'[color=blue][color=green][color=darkred]
>>> import threading
>>> s=socket.socket (socket.AF_INET ,socket.SOCK_ST REAM)
>>> s.bind(('127.0. 0.1',80))
>>> s.listen(1)
>>> t=threading.Thr ead(target=s.ac cept)
>>> t.start()
>>> t.isAlive()[/color][/color][/color]
True[color=blue][color=green][color=darkred]
>>> s.close()
>>> Exception in thread Thread-1:[/color][/color][/color]
Traceback (most recent call last):
File "C:\PYTHON23\li b\threading.py" , line 436, in __bootstrap
self.run()
File "C:\PYTHON23\li b\threading.py" , line 416, in run
self.__target(* self.__args, **self.__kwargs )
File "socket.py" , line 168, in accept
sock, addr = self._sock.acce pt()
error: (10038, 'Socket operation on non-socket')

[color=blue][color=green][color=darkred]
>>> t.isAlive()[/color][/color][/color]
False[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

i.e. close socket generates an exception on accept and no more connections.

If there's some underlying difficulty that causes this to come back and bite
the testing i've done shows this to take a very long time.

simon.

**Alex Martelli** · Jul 18 '05, 03:59 AM

Re: closing a server socket

simon place wrote:
...[color=blue]
> Yes, no problem closing a server before it returns to accept another
> connection, but when the handler is threaded,[/color]

When the server is threaded, it should be designed accordingly; see
Python's standard library, module SocketServer etc, for one right
way to design a threaded server.
[color=blue]
> i.e. closing server socket does not unblock accept and you can still get
> connections on it!
> ( because it has not actually been closed!)
> how is this acceptable?[/color]

It's not: in general, two threads should NEVER access the same
resource simultaneously, unless the resource is specifically
known to be thread safe. sockets aren't. It's not acceptable,
and the bug is in the program which has one of its threads
make a sock.close() call while another is waiting on a
sock.accept() on the same socket sock.

[color=blue]
> with socket modified to actually close the socket,[/color]
...[color=blue]
> i.e. close socket generates an exception on accept and no more
> connections.[/color]

On _your_ platform, no doubt. I'd bet substantial money that,
without even going to especially _exotic_ platforms, you'll find
terrible misbehavior with such mods on at least one of the
platforms I routinely program for.

[color=blue]
> If there's some underlying difficulty that causes this to come back and
> bite the testing i've done shows this to take a very long time.[/color]

Just as much time as is necessary to go and purchase some
other platform to try it on, I suspect.

Alex

closing a server socket

closing a server socket

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment