need a thread to keep a socket connection alive?

[nephish@xit.net]

hey there,

i have a script that waits for message packets from a data server over
a socket.
it goes a little like this:

while 1:
x+=1
databack = sockobj.recv(15 8)
if databack:

print 'caught a message %s bytes ' % len(databack)
if len(databack) > 120:
message = databack[3:-3] #strip stx and enx
print '\n\n%s' % message
else:
break
print 'end data ack'


it works fine for a while, but the server requires that i send a
heartbeat ping every 600 seconds or it will terminate the connection.

so i also need something like
while 1:
sockobj.send(pi ng)
ping_acknowlage = sockobj.recv(48 )
time.sleep(550)



should i do this with threads? i dont want to interrupt the listening
cycle to send a ping.

appreciate any tips on how would be the best way to pull this off.

[Rene Pijlman]

Re: need a thread to keep a socket connection alive?

nephish@xit.net:[color=blue]
>i have a script that waits for message packets from a data server over
>a socket.[/color]

Using what network protocol?
[color=blue]
>it works fine for a while, but the server requires that i send a
>heartbeat ping every 600 seconds or it will terminate the connection.[/color]
[...][color=blue]
>should i do this with threads? i dont want to interrupt the listening
>cycle to send a ping.[/color]

If this is a TCP connection with a conversational protocol, you can't have
two threads reading bytes of the socket, without some sort of
coordination. When one thread parses the bytes it received, some bytes may
be part of the next message for the other thread.

You may be better off with asynchronous I/O and a state machine model.

http://squirl.nightmare.com/medusa/async_sockets.html

http://www.python.org/doc/lib/module-asyncore.html

http://twistedmatrix.com/projects/core/documentation/howto/clients.html

--
René Pijlman

[nephish@xit.net]

Re: need a thread to keep a socket connection alive?

thanks for the info, i will likely use the first link you posted with
the async module just to get it going, but i want to learn more about
twisted for later. there is even an O'Reilly book on it i see.
thanks for the tips,
sk

[Ben Sizer]

Re: need a thread to keep a socket connection alive?

nephish@xit.net wrote:[color=blue]
> i have a script that waits for message packets from a data server over
> a socket.[/color]

If you're using TCP, bear in mind that you do not receive packets - you
receive a stream of data, which may usually come in the same quantities
as it was sent, but not always. If you don't take that into account,
you may end up missing a valid message because it arrived in several
parts.
[color=blue]
> it works fine for a while, but the server requires that i send a
> heartbeat ping every 600 seconds or it will terminate the connection.[/color]

It is probably worth just reconnecting if necessary. After all, you
could be disconnected for other reasons too.

--
Ben Sizer

[Roy Smith]

Re: need a thread to keep a socket connection alive?

nephish@xit.net wrote:
[color=blue]
> hey there,
>
> i have a script that waits for message packets from a data server over
> a socket.
> it goes a little like this:
>
> while 1:
> x+=1
> databack = sockobj.recv(15 8)
> if databack:
>
> print 'caught a message %s bytes ' % len(databack)
> if len(databack) > 120:
> message = databack[3:-3] #strip stx and enx
> print '\n\n%s' % message
> else:
> break
> print 'end data ack'[/color]

You need to go review how TCP works. All that it guarantees is that you
will receive bytes in the same order they were sent. It says nothing about
maintaining record boundaries. Just because you did a send(n) at one end,
it doesn't mean that you can expect to read n bytes in a single recv() call
at this end. Multiple send() calls could have their contents accumlated
into a single recv() call, or a single send() could get broken up into
several recv() calls.

If you want to read fixed-length messages (as you appear to be trying to do
with your recv(158)), you need to build a buffering layer which reads from
the socket into a buffer and then doles out messages to a higher layer from
that buffer.
[color=blue]
> it works fine for a while, but the server requires that i send a
> heartbeat ping every 600 seconds or it will terminate the connection.
>
> so i also need something like
> while 1:
> sockobj.send(pi ng)
> ping_acknowlage = sockobj.recv(48 )
> time.sleep(550)[/color]

This needs to be burried in a lower layer as well. You want to build some
kind of bufferedConnect ion class which hides all this gunk from your
application. You probably will want sendMessage() and recvMessage()
methods for your class. You probably want to have this class create a
thread to handle the low-level I/O asyncronously, and put the heatbeat
processing in there.

This is not a trivial problem. By the time you're done with it, you will
have learned a lot about how to communicate over a network.

[Fredrik Lundh]

Re: need a thread to keep a socket connection alive?

Roy Smith wrote:
[color=blue]
> If you want to read fixed-length messages (as you appear to be trying to do
> with your recv(158)), you need to build a buffering layer which reads from
> the socket into a buffer and then doles out messages to a higher layer from
> that buffer.[/color]
[color=blue]
> This is not a trivial problem. By the time you're done with it, you will
> have learned a lot about how to communicate over a network.[/color]

however, creating a buffered layer for reading is a trivial problem: just call
makefile on the socket object, and use the resulting object as a file handle:
[color=blue][color=green][color=darkred]
>>> s = socket.socket()
>>> s.connect(("www .python.org", 80))
>>> s.send("GET / HTTP/1.0\n\n")[/color][/color][/color]
16[color=blue][color=green][color=darkred]
>>> f = s.makefile()
>>> f.readline()[/color][/color][/color]
'HTTP/1.1 200 OK\r\n'[color=blue][color=green][color=darkred]
>>> f.readline()[/color][/color][/color]
'Date: Mon, 24 Apr 2006 12:37:46 GMT\r\n'[color=blue][color=green][color=darkred]
>>> f.read(10)[/color][/color][/color]
'Server: Ap'[color=blue][color=green][color=darkred]
>>> f.read(10)[/color][/color][/color]
'ache/2.0.5'[color=blue][color=green][color=darkred]
>>> f.readline()[/color][/color][/color]
'4 (Debian GNU/Linux) DAV/2 SVN/1.1.4 mod_python/3.1.3 ...[color=blue][color=green][color=darkred]
>>> f.readline()[/color][/color][/color]
'Last-Modified: Mon, 24 Apr 2006 04:52:53 GMT\r\n'

etc.

</F>

[Roy Smith]

Re: need a thread to keep a socket connection alive?

"Fredrik Lundh" <fredrik@python ware.com> wrote:[color=blue]
> however, creating a buffered layer for reading is a trivial problem: just call
> makefile on the socket object, and use the resulting object as a file handle:[/color]

The problem with that is that makefile() requires the socket to be in
blocking mode. If you're going to be implementing heartbeat, that's
probably not what you want.

[Serge Orlov]

Re: need a thread to keep a socket connection alive?

nephish@xit.net wrote:[color=blue]
> hey there,
>
> i have a script that waits for message packets from a data server over
> a socket.
> it goes a little like this:
>
> while 1:
> x+=1
> databack = sockobj.recv(15 8)
> if databack:
>
> print 'caught a message %s bytes ' % len(databack)
> if len(databack) > 120:
> message = databack[3:-3] #strip stx and enx
> print '\n\n%s' % message
> else:
> break
> print 'end data ack'
>
>
> it works fine for a while, but the server requires that i send a
> heartbeat ping every 600 seconds or it will terminate the connection.
>
> so i also need something like
> while 1:
> sockobj.send(pi ng)
> ping_acknowlage = sockobj.recv(48 )
> time.sleep(550)
>
>
>
> should i do this with threads? i dont want to interrupt the listening
> cycle to send a ping.
>
> appreciate any tips on how would be the best way to pull this off.[/color]

sockobj.settime out(550)

before the loop and later in the loop:

try:
databack = sockobj.recv(15 8)
except socket.timeout:
ping_server(soc kobj)
continue

Also, as other people pointed out, you'd better make buffered socket
with .makefile() socket method.

[Roy Smith]

Re: need a thread to keep a socket connection alive?

Serge Orlov <Serge.Orlov@gm ail.com> wrote:[color=blue]
> sockobj.settime out(550)
> [...]
> Also, as other people pointed out, you'd better make buffered socket
> with .makefile() socket method.[/color]

If I understand the docs for the socket module correctly, these two
suggestions are mutually incompatable.

[Serge Orlov]

Re: need a thread to keep a socket connection alive?

Roy Smith wrote:[color=blue]
> Serge Orlov <Serge.Orlov@gm ail.com> wrote:[color=green]
> > sockobj.settime out(550)
> > [...]
> > Also, as other people pointed out, you'd better make buffered socket
> > with .makefile() socket method.[/color]
>
> If I understand the docs for the socket module correctly, these two
> suggestions are mutually incompatable.[/color]

Perhaps this restriction was lifted?
[color=blue][color=green][color=darkred]
>>> s = socket.socket()
>>> s.settimeout(15 )
>>> s.connect(("www .python.org", 80))
>>> f = s.makefile()
>>> f.readline()[/color][/color][/color]

Traceback (most recent call last):
File "<pyshell#2 1>", line 1, in -toplevel-
f.readline()
File "C:\Python24\li b\socket.py", line 340, in readline
data = self._sock.recv (self._rbufsize )
timeout: timed out[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

[nephish@xit.net]

Re: need a thread to keep a socket connection alive?

ok, thanks for all the suggestions, gents, i clearly have more to read
on this.
i have discovered that the server will send a request for the heartbeat
ping if its almost timed out, so i use the length of the message to
determine what to do with it.

msg = sockobj.recv(10 24)

if len(msg) == 158:
record the data
elif len(msg) == (34): # length of request for ping
ping the server
else:
yada yada.

each real message ( according to their docs ) should be exactly 158
bytes.

i know i need to look more into all of this.. but thanks for all of
your help

-shawn

[Roy Smith]

Re: need a thread to keep a socket connection alive?

<nephish@xit.ne t> wrote:[color=blue]
>elif len(msg) == (34): # length of request for ping
> ping the server[/color]

This seems really dangerous. You are obviously writing to some
already defined (and hopefully, documented) protocol. What does the
protocol spec say about ping request messages?

I would be very surprised if the spec says that a ping request is
defined as any message which is 34 bytes long. More likely, it says
something like, "A ping request is defined by the command code being
set to 5" or something like that. That's what you should be testing,
not the length of the message.

[Rene Pijlman]

Re: need a thread to keep a socket connection alive?

nephish@xit.net:[color=blue]
>i have discovered that the server will send a request for the heartbeat
>ping if its almost timed out, so i use the length of the message to
>determine what to do with it.
>
>msg = sockobj.recv(10 24)
>
>if len(msg) == 158:
> record the data
>elif len(msg) == (34): # length of request for ping
> ping the server[/color]

Incorrect. There are no 'messages' with TCP, only bytes.

--
René Pijlman

[nephish@xit.net]

Re: need a thread to keep a socket connection alive?

ok, every message starts with "ENX" and ends with "STX"
in between are several parts. the first is the message length sent as
an unsigned long int (according to the docs) this is four bytes. The
next is the message type - another 4 bytes that corrospond to a certain
chart. for example, the login is message type 200. Then there is the
message. different messages have different lengths and types.
the data part, i am still trying to break apart. Some of it (when
printed to the screen) is a string, then some very funny characters,
then another string, then more funny looking characters, then 'ETX'

up untill now, all i have had to process is the strings, so i split the
message up, and used the strings as my data. But now, i do need the
other info. -whew.

i guess i am learning a lot. i just ordered Python Essential Reference
from amazon. ;)

anyway. from what i have found on line, i am going to be using the
struct module.



oh, that was a typo earlier. not 34 bytes, but 14.
the data comming in is alway in 158 bytes though.

thanks guys.

[Ben Sizer]

Re: need a thread to keep a socket connection alive?

nephish@xit.net wrote:[color=blue]
> the data comming in is alway in 158 bytes though.[/color]

And one day it may not. :) Consider yourself warned! (In a friendly
manner.)

--
Ben Sizer