UDP performance

[Paul Sijben]

I am stumped by the following problem. I have a large multi-threaded
server accepting communications on one UDP port (chosen for its supposed
speed).

I have been profiling the code and found that the UDP communication is
my biggest drain on performance! Communication where the client and the
server are on the same machine still takes 300ms or sometimes much more
per packet on an Athlon64 3000+ running Linux (Fedora Core 5 x64).

I must be doing something wrong and would really appreciate feedback on
my code below:

I open the server port with

self.s = socket.socket(s ocket.AF_INET, socket.SOCK_DGR AM)
self.s.setsocko pt(socket.SOL_S OCKET, socket.SO_REUSE ADDR, 1)
self.s.bind((my address, myport))

I then open a client port with

self.s=socket.s ocket(socket.AF _INET, socket.SOCK_DGR AM)
self.s.connect( (host, port))

the client sends data with

self.s.sendall( data)

and the server with

self.s.sendto(d ata,link.remote address)

both receive with

buf, address = socket.recvfrom (8192)

The sender and receiver are in separate threads (threading.Thre ad).

Does anyone know what is going wrong here, the socket communication, the
thread scheduling?

Paul Sijben

[Serge Orlov]

Re: UDP performance


Paul Sijben wrote:[color=blue]
> I am stumped by the following problem. I have a large multi-threaded
> server accepting communications on one UDP port (chosen for its supposed
> speed).
>
> I have been profiling the code and found that the UDP communication is
> my biggest drain on performance! Communication where the client and the
> server are on the same machine still takes 300ms or sometimes much more
> per packet on an Athlon64 3000+ running Linux (Fedora Core 5 x64).[/color]

[snip]
[color=blue]
> buf, address = socket.recvfrom (8192)[/color]

I'm not an expert here, but I AFAIK UDP packet size should be kept
below 512 bytes otherwise it can cause fragmentation and hence all the
slow stuff like acknoledgements , timeouts, etc...

[Paul Sijben]

Re: UDP performance

Serge Orlov wrote:[color=blue]
> Paul Sijben wrote:[color=green]
>> I am stumped by the following problem. I have a large multi-threaded
>> server accepting communications on one UDP port (chosen for its supposed
>> speed).
>>
>> I have been profiling the code and found that the UDP communication is
>> my biggest drain on performance! Communication where the client and the
>> server are on the same machine still takes 300ms or sometimes much more
>> per packet on an Athlon64 3000+ running Linux (Fedora Core 5 x64).[/color]
>
> [snip]
>[color=green]
>> buf, address = socket.recvfrom (8192)[/color]
>
> I'm not an expert here, but I AFAIK UDP packet size should be kept
> below 512 bytes otherwise it can cause fragmentation and hence all the
> slow stuff like acknoledgements , timeouts, etc...
>[/color]
good point in general but in practice they are in this case. but this is
less of a problem on the loopback interface anyway.

[Serge Orlov]

Re: UDP performance


Paul Sijben wrote:[color=blue]
> Serge Orlov wrote:[color=green]
> > Paul Sijben wrote:[color=darkred]
> >> I am stumped by the following problem. I have a large multi-threaded
> >> server accepting communications on one UDP port (chosen for its supposed
> >> speed).
> >>
> >> I have been profiling the code and found that the UDP communication is
> >> my biggest drain on performance! Communication where the client and the
> >> server are on the same machine still takes 300ms or sometimes much more
> >> per packet on an Athlon64 3000+ running Linux (Fedora Core 5 x64).[/color]
> >
> > [snip]
> >[color=darkred]
> >> buf, address = socket.recvfrom (8192)[/color]
> >
> > I'm not an expert here, but I AFAIK UDP packet size should be kept
> > below 512 bytes otherwise it can cause fragmentation and hence all the
> > slow stuff like acknoledgements , timeouts, etc...
> >[/color]
> good point in general but in practice they are in this case. but this is
> less of a problem on the loopback interface anyway.[/color]

Isn't it still controlled by MTU even on the loopback? What is your MTU
on the loopback?

[Paul Sijben]

Re: UDP performance

Serge Orlov wrote:[color=blue]
> Paul Sijben wrote:[color=green]
>> Serge Orlov wrote:[color=darkred]
>>> Paul Sijben wrote:
>>>> I am stumped by the following problem. I have a large multi-threaded
>>>> server accepting communications on one UDP port (chosen for its supposed
>>>> speed).
>>>>
>>>> I have been profiling the code and found that the UDP communication is
>>>> my biggest drain on performance! Communication where the client and the
>>>> server are on the same machine still takes 300ms or sometimes much more
>>>> per packet on an Athlon64 3000+ running Linux (Fedora Core 5 x64).
>>> [snip]
>>>
>>>> buf, address = socket.recvfrom (8192)
>>> I'm not an expert here, but I AFAIK UDP packet size should be kept
>>> below 512 bytes otherwise it can cause fragmentation and hence all the
>>> slow stuff like acknoledgements , timeouts, etc...
>>>[/color]
>> good point in general but in practice they are in this case. but this is
>> less of a problem on the loopback interface anyway.[/color]
>
> Isn't it still controlled by MTU even on the loopback? What is your MTU
> on the loopback?
>[/color]
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1

eth0 Link encap:Ethernet HWaddr 00:14:85:35:A4: 5D
inet addr:192.168.0. 124 Bcast:192.168.0 .255 Mask:255.255.25 5.0
inet6 addr: fe80::214:85ff: fe35:a45d/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

[Martin P. Hellwig]

Re: UDP performance

Paul Sijben wrote:[color=blue]
> I am stumped by the following problem. I have a large multi-threaded
> server accepting communications on one UDP port (chosen for its supposed
> speed).
>
> I have been profiling the code and found that the UDP communication is
> my biggest drain on performance! Communication where the client and the
> server are on the same machine still takes 300ms or sometimes much more
> per packet on an Athlon64 3000+ running Linux (Fedora Core 5 x64).
>
> I must be doing something wrong and would really appreciate feedback on
> my code below:
>
> I open the server port with
>
> self.s = socket.socket(s ocket.AF_INET, socket.SOCK_DGR AM)
> self.s.setsocko pt(socket.SOL_S OCKET, socket.SO_REUSE ADDR, 1)
> self.s.bind((my address, myport))
>
> I then open a client port with
>
> self.s=socket.s ocket(socket.AF _INET, socket.SOCK_DGR AM)
> self.s.connect( (host, port))
>
> the client sends data with
>
> self.s.sendall( data)
>
> and the server with
>
> self.s.sendto(d ata,link.remote address)
>
> both receive with
>
> buf, address = socket.recvfrom (8192)
>
> The sender and receiver are in separate threads (threading.Thre ad).
>
> Does anyone know what is going wrong here, the socket communication, the
> thread scheduling?
>
> Paul Sijben
>[/color]

Is the connection 1:1 i.e. the receiving end receives data only from one
sender at the time? And how do you handle lost packages in your application?

--
mph

[Paul Sijben]

Re: UDP performance

Martin P. Hellwig wrote:
[snip][color=blue]
> Is the connection 1:1 i.e. the receiving end receives data only from one
> sender at the time? And how do you handle lost packages in your
> application?
>[/color]

no the server is receiving from multiple clients.

and at the moment I do not handle lost packets.

[Paul Sijben]

massive threading performance (was:Re: UDP performance)

OK the problem I posted about earlier is NOT a UDP/socket problem, it is
a threading problem. Albeit one that only happens when you have many
thrreads????

I have made two little scripts using the code I copied below, one client
and one server (attached).

If I run them concurrently on the same machine I see the following times:[color=blue][color=green]
>>1145526192.82 5508[/color][/color]
<<1145526192.82 5848
[color=blue][color=green]
>>1145526193.82 9325[/color][/color]
<<1145526193.83 4927

a transfer time in the milliseconds. Much less than the times I see in
the full application.

OK so I put them both in a multithreaded script (also attached)

[color=blue][color=green]
>>1145526971.61 9558[/color][/color]
<<1145526971.61 9909
[color=blue][color=green]
>>1145526972.61 9241[/color][/color]
<<1145526972.61 9647

again transfer time in milliseconds.

Not like this that I get from the profile of my code:
<< 1145517554.3638 50 send[color=blue][color=green]
>> 1145517554.6474 85 recv[/color][/color]

which uses the same communication and threading but has 20+ threads?

Now I am completely baffled!

again I really appreciate if anyone can shed light on this!

Paul


Paul Sijben wrote:[color=blue]
> I am stumped by the following problem. I have a large multi-threaded
> server accepting communications on one UDP port (chosen for its supposed
> speed).
>
> I have been profiling the code and found that the UDP communication is
> my biggest drain on performance! Communication where the client and the
> server are on the same machine still takes 300ms or sometimes much more
> per packet on an Athlon64 3000+ running Linux (Fedora Core 5 x64).
>
> I must be doing something wrong and would really appreciate feedback on
> my code below:
>
> I open the server port with
>
> self.s = socket.socket(s ocket.AF_INET, socket.SOCK_DGR AM)
> self.s.setsocko pt(socket.SOL_S OCKET, socket.SO_REUSE ADDR, 1)
> self.s.bind((my address, myport))
>
> I then open a client port with
>
> self.s=socket.s ocket(socket.AF _INET, socket.SOCK_DGR AM)
> self.s.connect( (host, port))
>
> the client sends data with
>
> self.s.sendall( data)
>
> and the server with
>
> self.s.sendto(d ata,link.remote address)
>
> both receive with
>
> buf, address = socket.recvfrom (8192)
>
> The sender and receiver are in separate threads (threading.Thre ad).
>
> Does anyone know what is going wrong here, the socket communication, the
> thread scheduling?
>
> Paul Sijben
>[/color]



import socket
import time

profile=[]

s=socket.socket (socket.AF_INET , socket.SOCK_DGR AM)
s.connect(('127 .0.0.1', 42420))
print ">>%f"%time.tim e()
s.sendall('test erdetest')
time.sleep(1)
print ">>%f"%time.tim e()
s.sendall('test erdetest')
print ">>%f"%time.tim e()


import socket
import time

profile=[]


s = socket.socket(s ocket.AF_INET, socket.SOCK_DGR AM)
s.setsockopt(so cket.SOL_SOCKET , socket.SO_REUSE ADDR, 1)
s.bind(('', 42420))


buf, address =s.recvfrom(819 2)
print "<<%f"%time.tim e()
print buf, address
buf, address =s.recvfrom(819 2)
print "<<%f"%time.tim e()
print buf, address

import socket
import time
import threading
profile=[]

class server(threadin g.Thread):
def __init__(self):
self.s = socket.socket(s ocket.AF_INET, socket.SOCK_DGR AM)
self.s.setsocko pt(socket.SOL_S OCKET, socket.SO_REUSE ADDR, 1)
self.s.bind(('' , 42420))
threading.Threa d.__init__(self )
self.start()

def run(self):
buf, address =self.s.recvfro m(8192)
print "<<%f"%time.tim e()
print buf, address
buf, address =self.s.recvfro m(8192)
print "<<%f"%time.tim e()
print buf, address

class client(threadin g.Thread):
def __init__(self):
self.s =socket.socket( socket.AF_INET, socket.SOCK_DGR AM)
self.s.connect( ('127.0.0.1', 42420))
threading.Threa d.__init__(self )
self.start()

def run(self):
print ">>%f"%time.tim e()
self.s.sendall( 'testerdetest')
time.sleep(1)
print ">>%f"%time.tim e()
self.s.sendall( 'testerdetest')
print ">>%f"%time.tim e()


serv=server()
cl=client()

[Paul Sijben]

FOUNDIT (was Re: massive threading performance)

I found that the problem was caused by the sending thread not giving
control back quickly enough to the receiving thread.

Also in going through the code I found an old self.s.setblock ing(0)call
that was no longer relevant. Removing that solved my problem.

Something that took 20 seconds now takes just 1.

Thanks to those who took the time to respond to my earlier messages.

Paul

Paul Sijben wrote:[color=blue]
> OK the problem I posted about earlier is NOT a UDP/socket problem, it is
> a threading problem. Albeit one that only happens when you have many
> thrreads????
>
> I have made two little scripts using the code I copied below, one client
> and one server (attached).
>
> If I run them concurrently on the same machine I see the following times:[color=green][color=darkred]
>>> 1145526192.8255 08[/color][/color]
> <<1145526192.82 5848
>[color=green][color=darkred]
>>> 1145526193.8293 25[/color][/color]
> <<1145526193.83 4927
>
> a transfer time in the milliseconds. Much less than the times I see in
> the full application.
>
> OK so I put them both in a multithreaded script (also attached)
>
>[color=green][color=darkred]
>>> 1145526971.6195 58[/color][/color]
> <<1145526971.61 9909
>[color=green][color=darkred]
>>> 1145526972.6192 41[/color][/color]
> <<1145526972.61 9647
>
> again transfer time in milliseconds.
>
> Not like this that I get from the profile of my code:
> << 1145517554.3638 50 send[color=green][color=darkred]
>>> 1145517554.6474 85 recv[/color][/color]
>
> which uses the same communication and threading but has 20+ threads?
>
> Now I am completely baffled!
>
> again I really appreciate if anyone can shed light on this!
>
> Paul
>
>
> Paul Sijben wrote:[color=green]
>> I am stumped by the following problem. I have a large multi-threaded
>> server accepting communications on one UDP port (chosen for its supposed
>> speed).
>>
>> I have been profiling the code and found that the UDP communication is
>> my biggest drain on performance! Communication where the client and the
>> server are on the same machine still takes 300ms or sometimes much more
>> per packet on an Athlon64 3000+ running Linux (Fedora Core 5 x64).
>>
>> I must be doing something wrong and would really appreciate feedback on
>> my code below:
>>
>> I open the server port with
>>
>> self.s = socket.socket(s ocket.AF_INET, socket.SOCK_DGR AM)
>> self.s.setsocko pt(socket.SOL_S OCKET, socket.SO_REUSE ADDR, 1)
>> self.s.bind((my address, myport))
>>
>> I then open a client port with
>>
>> self.s=socket.s ocket(socket.AF _INET, socket.SOCK_DGR AM)
>> self.s.connect( (host, port))
>>
>> the client sends data with
>>
>> self.s.sendall( data)
>>
>> and the server with
>>
>> self.s.sendto(d ata,link.remote address)
>>
>> both receive with
>>
>> buf, address = socket.recvfrom (8192)
>>
>> The sender and receiver are in separate threads (threading.Thre ad).
>>
>> Does anyone know what is going wrong here, the socket communication, the
>> thread scheduling?
>>
>> Paul Sijben
>>[/color]
>
>
>
> ------------------------------------------------------------------------
>
> import socket
> import time
>
> profile=[]
>
> s=socket.socket (socket.AF_INET , socket.SOCK_DGR AM)
> s.connect(('127 .0.0.1', 42420))
> print ">>%f"%time.tim e()
> s.sendall('test erdetest')
> time.sleep(1)
> print ">>%f"%time.tim e()
> s.sendall('test erdetest')
> print ">>%f"%time.tim e()
>
>
>
> ------------------------------------------------------------------------
>
> import socket
> import time
>
> profile=[]
>
>
> s = socket.socket(s ocket.AF_INET, socket.SOCK_DGR AM)
> s.setsockopt(so cket.SOL_SOCKET , socket.SO_REUSE ADDR, 1)
> s.bind(('', 42420))
>
>
> buf, address =s.recvfrom(819 2)
> print "<<%f"%time.tim e()
> print buf, address
> buf, address =s.recvfrom(819 2)
> print "<<%f"%time.tim e()
> print buf, address
>
>
> ------------------------------------------------------------------------
>
> import socket
> import time
> import threading
> profile=[]
>
> class server(threadin g.Thread):
> def __init__(self):
> self.s = socket.socket(s ocket.AF_INET, socket.SOCK_DGR AM)
> self.s.setsocko pt(socket.SOL_S OCKET, socket.SO_REUSE ADDR, 1)
> self.s.bind(('' , 42420))
> threading.Threa d.__init__(self )
> self.start()
>
> def run(self):
> buf, address =self.s.recvfro m(8192)
> print "<<%f"%time.tim e()
> print buf, address
> buf, address =self.s.recvfro m(8192)
> print "<<%f"%time.tim e()
> print buf, address
>
> class client(threadin g.Thread):
> def __init__(self):
> self.s =socket.socket( socket.AF_INET, socket.SOCK_DGR AM)
> self.s.connect( ('127.0.0.1', 42420))
> threading.Threa d.__init__(self )
> self.start()
>
> def run(self):
> print ">>%f"%time.tim e()
> self.s.sendall( 'testerdetest')
> time.sleep(1)
> print ">>%f"%time.tim e()
> self.s.sendall( 'testerdetest')
> print ">>%f"%time.tim e()
>
>
> serv=server()
> cl=client()[/color]

[Paul Sijben]

massive threading performance (was:Re: UDP performance)

OK the problem I posted about earlier is NOT a UDP/socket problem, it is
a threading problem. Albeit one that only happens when you have many
thrreads????

I have made two little scripts using the code I copied below, one client
and one server (attached).

If I run them concurrently on the same machine I see the following times:[color=blue][color=green]
>>1145526192.82 5508[/color][/color]
<<1145526192.82 5848
[color=blue][color=green]
>>1145526193.82 9325[/color][/color]
<<1145526193.83 4927

a transfer time in the milliseconds. Much less than the times I see in
the full application.

OK so I put them both in a multithreaded script (also attached)

[color=blue][color=green]
>>1145526971.61 9558[/color][/color]
<<1145526971.61 9909
[color=blue][color=green]
>>1145526972.61 9241[/color][/color]
<<1145526972.61 9647

again transfer time in milliseconds.

Not like this that I get from the profile of my code:
<< 1145517554.3638 50 send[color=blue][color=green]
>> 1145517554.6474 85 recv[/color][/color]

which uses the same communication and threading but has 20+ threads?

Now I am completely baffled!

again I really appreciate if anyone can shed light on this!

Paul


Paul Sijben wrote:[color=blue]
> I am stumped by the following problem. I have a large multi-threaded
> server accepting communications on one UDP port (chosen for its supposed
> speed).
>
> I have been profiling the code and found that the UDP communication is
> my biggest drain on performance! Communication where the client and the
> server are on the same machine still takes 300ms or sometimes much more
> per packet on an Athlon64 3000+ running Linux (Fedora Core 5 x64).
>
> I must be doing something wrong and would really appreciate feedback on
> my code below:
>
> I open the server port with
>
> self.s = socket.socket(s ocket.AF_INET, socket.SOCK_DGR AM)
> self.s.setsocko pt(socket.SOL_S OCKET, socket.SO_REUSE ADDR, 1)
> self.s.bind((my address, myport))
>
> I then open a client port with
>
> self.s=socket.s ocket(socket.AF _INET, socket.SOCK_DGR AM)
> self.s.connect( (host, port))
>
> the client sends data with
>
> self.s.sendall( data)
>
> and the server with
>
> self.s.sendto(d ata,link.remote address)
>
> both receive with
>
> buf, address = socket.recvfrom (8192)
>
> The sender and receiver are in separate threads (threading.Thre ad).
>
> Does anyone know what is going wrong here, the socket communication, the
> thread scheduling?
>
> Paul Sijben
>[/color]


import socket
import time

profile=[]

s=socket.socket (socket.AF_INET , socket.SOCK_DGR AM)
s.connect(('127 .0.0.1', 42420))
print ">>%f"%time.tim e()
s.sendall('test erdetest')
time.sleep(1)
print ">>%f"%time.tim e()
s.sendall('test erdetest')
print ">>%f"%time.tim e()

import socket
import time

profile=[]


s = socket.socket(s ocket.AF_INET, socket.SOCK_DGR AM)
s.setsockopt(so cket.SOL_SOCKET , socket.SO_REUSE ADDR, 1)
s.bind(('', 42420))


buf, address =s.recvfrom(819 2)
print "<<%f"%time.tim e()
print buf, address
buf, address =s.recvfrom(819 2)
print "<<%f"%time.tim e()
print buf, address
import socket
import time
import threading
profile=[]

class server(threadin g.Thread):
def __init__(self):
self.s = socket.socket(s ocket.AF_INET, socket.SOCK_DGR AM)
self.s.setsocko pt(socket.SOL_S OCKET, socket.SO_REUSE ADDR, 1)
self.s.bind(('' , 42420))
threading.Threa d.__init__(self )
self.start()

def run(self):
buf, address =self.s.recvfro m(8192)
print "<<%f"%time.tim e()
print buf, address
buf, address =self.s.recvfro m(8192)
print "<<%f"%time.tim e()
print buf, address

class client(threadin g.Thread):
def __init__(self):
self.s =socket.socket( socket.AF_INET, socket.SOCK_DGR AM)
self.s.connect( ('127.0.0.1', 42420))
threading.Threa d.__init__(self )
self.start()

def run(self):
print ">>%f"%time.tim e()
self.s.sendall( 'testerdetest')
time.sleep(1)
print ">>%f"%time.tim e()
self.s.sendall( 'testerdetest')
print ">>%f"%time.tim e()


serv=server()
cl=client()

[Lawrence D'Oliveiro]

Re: FOUNDIT (was Re: massive threading performance)

In article <44476e82$0$316 50$e4fe514c@new s.xs4all.nl>,
Paul Sijben <sijben@eemvall ey.com> wrote:
[color=blue]
>I found that the problem was caused by the sending thread not giving
>control back quickly enough to the receiving thread.
>
>Also in going through the code I found an old self.s.setblock ing(0)call
>that was no longer relevant. Removing that solved my problem.
>
>Something that took 20 seconds now takes just 1.[/color]

You might also find that it goes still faster if you forego threading
and use a select.select loop.

[Paul Sijben]

Re: FOUNDIT (was Re: massive threading performance)

Lawrence D'Oliveiro wrote:[color=blue]
> In article <44476e82$0$316 50$e4fe514c@new s.xs4all.nl>,
> Paul Sijben <sijben@eemvall ey.com> wrote:
>[color=green]
>> I found that the problem was caused by the sending thread not giving
>> control back quickly enough to the receiving thread.
>>
>> Also in going through the code I found an old self.s.setblock ing(0)call
>> that was no longer relevant. Removing that solved my problem.
>>
>> Something that took 20 seconds now takes just 1.[/color]
>
> You might also find that it goes still faster if you forego threading
> and use a select.select loop.[/color]

thanks for that. I will have to try it.