Socket Performance

Collapse
This topic is closed.
X
X
 
  • Time
  • Show
Clear All
new posts
  • sleddd@gmail.com

    Socket Performance

    Can anyone explain why socket performance (throughput) varies
    depending on the amount of data send and recv are called with?

    For example: try creating a local client/server (running on the same
    computer) where the server sends the client a fixed amount of data.
    Using method A, recv(8192) and sendall( ) with 8192 bytes worth of
    data. Do this 100 times. Using method B, recv(1) and sendall( ) with 1
    byte worth of data. Do this 819200 times.

    If you time both methods, method A has much greater throughput than
    method B.

    Server:

    import socket
    import random
    import string
    import time

    HOST = 'localhost'
    PORT = 50023
    s = socket.socket(s ocket.AF_INET, socket.SOCK_STR EAM)
    s.bind((HOST, PORT))
    s.listen(1)
    conn, addr = s.accept()
    print 'Connected by', addr

    string_1 = 'A'
    string_8192 = ''.join([random.choice(s tring.letters + string.digits)
    for i in range(8192)])

    conn.sendall('S tart')

    start = time.clock()
    total_data = 0
    for i in range(0,100):
    conn.sendall(st ring_8192)
    total_data += len(string_8192 )
    print 'Send Speed (Long String): ' + str( total_data / (time.clock() -
    start) / 1024 / 1024 ) + ' MB/sec\n\n'

    start = time.clock()
    total_data = 0
    for i in range(0,819200) :
    conn.sendall(st ring_1)
    total_data += len(string_1)
    print 'Send Speed (Short String): ' + str( total_data / (time.clock()
    - start) / 1024 / 1024 ) + ' MB/sec'

    conn.close()


    Client:

    import socket
    import time

    HOST = 'localhost' # The remote host
    PORT = 50023 # The same port as used by the server
    s = socket.socket(s ocket.AF_INET, socket.SOCK_STR EAM)
    s.bind((HOST, 50026))
    s.connect((HOST , PORT))

    data = s.recv(5)
    print 'From Server: ' + data

    start = time.clock()
    total_data = 0
    while total_data < 819200:
    data = s.recv(8192)
    total_data += len(data)
    print 'Receive Speed (Long String): ' + str( total_data /
    (time.clock() - start) / 1024 / 1024 ) + ' MB/sec\n\n'

    start = time.clock()
    total_data = 0
    while total_data < 819200:
    data = s.recv(1)
    total_data += len(data)
    print 'Receive Speed (Short String): ' + str( total_data /
    (time.clock() - start) / 1024 / 1024 ) + ' MB/sec'
    s.close()
  • Brian Smith

    #2
    RE: Socket Performance

    sleddd@gmail.co m wrote:
    Sent: Wednesday, March 12, 2008 9:47 PM
    To: python-list@python.org
    Subject: Socket Performance
    >
    Can anyone explain why socket performance (throughput) varies
    depending on the amount of data send and recv are called with?
    >
    For example: try creating a local client/server (running on the same
    computer) where the server sends the client a fixed amount of data.
    Using method A, recv(8192) and sendall( ) with 8192 bytes
    worth of data. Do this 100 times. Using method B, recv(1) and
    sendall( ) with 1 byte worth of data. Do this 819200 times.
    >
    If you time both methods, method A has much greater
    throughput than method B.
    Why is it faster to drink a liter of water a cupful at a time than to
    drink it out of an eyedropper?

    - Brian

    Comment

    • Bryan Olson

      #3
      Re: Socket Performance

      castironpi@gmai l.com wrote:
      Gabriel Genellina wrote:
      >No need to reinvent the wheel. socket objects already have a makefile
      >method returning a file-like object, which behaves like a buffered socket.
      That wheel is far from round, and needs some reinvention. Python's
      file-like objects do not play nice with lower level calls, which
      would be tolerable if they supported some well-defiend high-level
      asynchronous I/O, but they do not.
      Newbie question: Can you write to the 'file-like object' a pickle,
      and receive it intact-- as one string with nothing else?
      Yes, but there's a world of gotcha's. Sockets do not recognize
      record boundaries, and Python's 'pickle' has holes one's enemies
      could drive a truck through. Still, you can pickle, write, read,
      un-pickle, and get back your data intact.
      I want to know because I want to send two pickles.
      "Two pickles" sounds like a tasty snack, but also suggests you may
      be playing hopscotch in a minefield. This is a helpful group. Give
      us more to go on, and you are likely to receive thousands of
      dollars worth of consulting for free.


      --
      --Bryan

      Comment

      • castironpi@gmail.com

        #4
        Re: Socket Performance

        On Mar 15, 8:18 am, Bryan Olson <fakeaddr...@no where.orgwrote:
        castiro...@gmai l.com wrote:
        Gabriel Genellina wrote:
        No need to reinvent the wheel. socket objects already have a makefile  
        method returning a file-like object, which behaves like a buffered socket.
        >
        That wheel is far from round, and needs some reinvention. Python's
        file-like objects do not play nice with lower level calls, which
        would be tolerable if they supported some well-defiend high-level
        asynchronous I/O, but they do not.
        >
        Newbie question:  Can you write to the 'file-like object' a pickle,
        and receive it intact-- as one string with nothing else?
        >
        Yes, but there's a world of gotcha's. Sockets do not recognize
        record boundaries, and Python's 'pickle' has holes one's enemies
        could drive a truck through. Still, you can pickle, write, read,
        un-pickle, and get back your data intact.
        >
        I want to know because I want to send two pickles.
        >
        "Two pickles" sounds like a tasty snack, but also suggests you may
        be playing hopscotch in a minefield. This is a helpful group. Give
        us more to go on, and you are likely to receive thousands of
        dollars worth of consulting for free.
        It depends on the situation. How generally applicable is this:

        fun ListenerGeneric ( port, factory, arrivedfun ):

        which calls 'factory' on socketA.accept (and loops again), then
        arrivedfun( stringA ) on message complete detection. ?. It should
        start itself in a separate thread.

        Or is this any better:

        for x in connections():
        startnewthreadw ith x:
        for y in messages( x ):
        arrivedfun( y )

        Comment

        • Gabriel Genellina

          #5
          Re: Socket Performance

          En Sat, 15 Mar 2008 20:08:05 -0200, <castironpi@gma il.comescribiï¿ ½:
          On Mar 15, 8:18 am, Bryan Olson <fakeaddr...@no where.orgwrote:
          >castiro...@gma il.com wrote:
          Newbie question:  Can you write to the 'file-like object' a pickle,
          and receive it intact-- as one string with nothing else?
          >>
          >Yes, but there's a world of gotcha's. Sockets do not recognize
          >record boundaries, and Python's 'pickle' has holes one's enemies
          >could drive a truck through. Still, you can pickle, write, read,
          >un-pickle, and get back your data intact.
          >>
          I want to know because I want to send two pickles.
          >>
          >"Two pickles" sounds like a tasty snack, but also suggests you may
          >be playing hopscotch in a minefield. This is a helpful group. Give
          >us more to go on, and you are likely to receive thousands of
          >dollars worth of consulting for free.
          >
          It depends on the situation. How generally applicable is this:
          >
          fun ListenerGeneric ( port, factory, arrivedfun ):
          >
          which calls 'factory' on socketA.accept (and loops again), then
          arrivedfun( stringA ) on message complete detection. ?. It should
          start itself in a separate thread.
          >
          Or is this any better:
          >
          for x in connections():
          startnewthreadw ith x:
          for y in messages( x ):
          arrivedfun( y )
          This looks like a SocketServer + ThreadingMixIn + a RequestHandler (your
          factory).
          But as B. Olson already pointed, pickles are unsafe. Worse, it's not that
          someone could send a specially crafted pickle that could execute some
          arbitrary code: you're blindy executing whatever you receive!
          xmlrpc may be a good alternative in some cases.

          --
          Gabriel Genellina

          Comment

          • castironpi@gmail.com

            #6
            Re: Socket Performance

            On Mar 16, 1:29 pm, "Gabriel Genellina" <gagsl-...@yahoo.com.a r>
            wrote:
            En Sat, 15 Mar 2008 20:08:05 -0200, <castiro...@gma il.comescribiï¿ ½:
            >
            >
            >
            >
            >
            On Mar 15, 8:18 am, Bryan Olson <fakeaddr...@no where.orgwrote:
            castiro...@gmai l.com wrote:
            Newbie question:  Can you write to the 'file-like object' a pickle,
            and receive it intact-- as one string with nothing else?
            >
            Yes, but there's a world of gotcha's. Sockets do not recognize
            record boundaries, and Python's 'pickle' has holes one's enemies
            could drive a truck through. Still, you can pickle, write, read,
            un-pickle, and get back your data intact.
            >
            I want to know because I want to send two pickles.
            >
            "Two pickles" sounds like a tasty snack, but also suggests you may
            be playing hopscotch in a minefield. This is a helpful group. Give
            us more to go on, and you are likely to receive thousands of
            dollars worth of consulting for free.
            >
            It depends on the situation.  How generally applicable is this:
            >
            fun ListenerGeneric ( port, factory, arrivedfun ):
            >
            which calls 'factory' on socketA.accept (and loops again), then
            arrivedfun( stringA ) on message complete detection.  ?.  It should
            start itself in a separate thread.
            >
            Or is this any better:
            >
            for x in connections():
               startnewthreadw ith x:
                  for y in messages( x ):
                     arrivedfun( y )
            >
            This looks like a SocketServer + ThreadingMixIn + a RequestHandler (your  
            factory).
            But as B. Olson already pointed, pickles are unsafe. Worse, it's not that  
            someone could send a specially crafted pickle that could execute some  
            arbitrary code: you're blindy executing whatever you receive!
            xmlrpc may be a good alternative in some cases.
            I see. Just transmit a binary and execute that. Make sure to have
            handles to your program publicly accessible too. I execute this.

            Comment

            Working...