socket.makefile() buggy?

Collapse
This topic is closed.
X
X
 
  • Time
  • Show
Clear All
new posts
  • ahlongxp

    #1

    socket.makefile() buggy?

    socket.makefile () may lose data when "connection reset by peer".
    and socket.recv() will never lose the data.

    change the "1" to "0" in the client code to see the difference.

    confirmed on both windows and linux.

    so I guess there is a problem with makefile().

    # Echo server program
    import socket

    HOST = '' # Symbolic name meaning the local host
    PORT = 50007 # Arbitrary non-privileged port
    s = socket.socket(s ocket.AF_INET, socket.SOCK_STR EAM)
    s.setsockopt(so cket.SOL_SOCKET , socket.SO_REUSE ADDR, 1)
    s.bind((HOST, PORT))
    s.listen(1)
    while(1):
    conn, addr = s.accept()
    print 'Connected by', addr
    conn.settimeout (1)
    toread = 99
    retrytime = 0
    reads = 0
    while reads < toread and retrytime < 10:
    try:
    data = conn.recv(min(3 2,toread-reads))
    if not data: continue
    print data
    reads += len(data)
    except:
    retrytime += 1
    print "timeout %d" % retrytime
    continue
    #conn.shutdown( socket.SHUT_RD) #no more read
    if reads == toread:
    conn.send("OK")
    else:
    conn.send("NOT OK")
    conn.close()

    # Echo client program
    import socket

    HOST = 'localhost' # The remote host
    PORT = 50007 # The same port as used by the server
    s = socket.socket(s ocket.AF_INET, socket.SOCK_STR EAM)
    s.connect((HOST , PORT))
    for i in range(12):
    print "time %d" % i
    try:
    s.send('0123456 789')
    except socket.error, e:
    print "socket error:", e
    break
    #data = s.recv(1024)
    #print "data %d" %i, data
    #s.shutdown(soc ket.SHUT_WR)#no more write

    '''
    try changing 1 to 0.
    '''
    if 1:
    data=s.recv(102 4)
    else:
    rf = s.makefile("rb" )
    data = rf.read()
    rf.close()
    s.close()
    print 'Received', repr(data)

  • Steve Holden

    #2
    Re: socket.makefile () buggy?

    That's a pretty pejorative subject line for someone who's been
    programming Python [guessing by the date of your first post] for about a
    month.

    Perhaps "Incomprehensib le behavior from socket.makefile ()", or "I have
    written a buggy network application"? That would at least show that you
    are considering the possibility you yourself are the cause of the
    problem ;-)

    Python has been around for a long time, so you should ask yourself how
    likely it is that you would be the first to discover such a fundamental
    flaw? I'd be very surprised if someone doesn't point you at an article
    on "how to ask smart questions", but I will content myself with that
    oblique reference.

    ahlongxp wrote:
    socket.makefile () may lose data when "connection reset by peer".
    and socket.recv() will never lose the data.
    >
    change the "1" to "0" in the client code to see the difference.
    >
    confirmed on both windows and linux.
    >
    so I guess there is a problem with makefile().
    >
    # Echo server program
    import socket
    >
    HOST = '' # Symbolic name meaning the local host
    PORT = 50007 # Arbitrary non-privileged port
    s = socket.socket(s ocket.AF_INET, socket.SOCK_STR EAM)
    s.setsockopt(so cket.SOL_SOCKET , socket.SO_REUSE ADDR, 1)
    s.bind((HOST, PORT))
    s.listen(1)
    while(1):
    conn, addr = s.accept()
    print 'Connected by', addr
    conn.settimeout (1)
    toread = 99
    retrytime = 0
    reads = 0
    while reads < toread and retrytime < 10:
    try:
    data = conn.recv(min(3 2,toread-reads))
    if not data: continue
    print data
    reads += len(data)
    except:
    retrytime += 1
    print "timeout %d" % retrytime
    continue
    #conn.shutdown( socket.SHUT_RD) #no more read
    if reads == toread:
    conn.send("OK")
    else:
    conn.send("NOT OK")
    conn.close()
    >
    # Echo client program
    import socket
    >
    HOST = 'localhost' # The remote host
    PORT = 50007 # The same port as used by the server
    s = socket.socket(s ocket.AF_INET, socket.SOCK_STR EAM)
    s.connect((HOST , PORT))
    for i in range(12):
    print "time %d" % i
    try:
    s.send('0123456 789')
    except socket.error, e:
    print "socket error:", e
    break
    #data = s.recv(1024)
    #print "data %d" %i, data
    #s.shutdown(soc ket.SHUT_WR)#no more write
    >
    '''
    try changing 1 to 0.
    '''
    if 1:
    data=s.recv(102 4)
    else:
    rf = s.makefile("rb" )
    data = rf.read()
    rf.close()
    s.close()
    print 'Received', repr(data)
    >
    >
    The big problem here seems to be that your server is closing the socket
    after reading 99 bytes, but the client is actually sending 120. If you
    change the "99" in the server to "120" you will see that the client
    works when it uses the makefile().read (). I can't be bothered to debug
    the exact reason why the 21 bytes of buffered data doesn't cause a
    problem with the recv() version, but I wouldn't regard the error you are
    getting as a bug in Python, rather as a bug in your application.

    The underlying cause of all this appears to be your failure to
    understand that network protocols should ideally allow the endpoints to
    determine when a complete message has been transmitted, and to consume
    all transmitted data.

    You can't just ignore some or all of a client request and expect it not
    to cause problems. If you look at some of the better-known TCP-based
    application protocols you will see they take pains to ensure that
    clients and servers can deterministical ly parse each others' messages.

    In summary, a better protocol design will cause you rather less grief
    and a little more humility will result in less acidity from crabby old
    geeks like me.

    regards
    Steve
    --
    Steve Holden +1 571 484 6266 +1 800 494 3119
    Holden Web LLC/Ltd http://www.holdenweb.com
    Skype: holdenweb http://del.icio.us/steve.holden
    --------------- Asciimercial ------------------
    Get on the web: Blog, lens and tag the Internet
    Many services currently offer free registration
    ----------- Thank You for Reading -------------

    Comment

    • Paul McGuire

      #3
      Re: socket.makefile () buggy?

      On Jul 8, 8:54 am, Steve Holden <s...@holdenweb .comwrote:
      That's a pretty pejorative subject line for someone who's been
      programming Python [guessing by the date of your first post] for about a
      month.
      >
      Perhaps "Incomprehensib le behavior from socket.makefile ()", or "I have
      written a buggy network application"? That would at least show that you
      are considering the possibility you yourself are the cause of the
      problem ;-)
      >
      Python has been around for a long time, so you should ask yourself how
      likely it is that you would be the first to discover such a fundamental
      flaw? I'd be very surprised if someone doesn't point you at an article
      on "how to ask smart questions", but I will content myself with that
      oblique reference.
      >
      >
      >
      >
      >
      ahlongxp wrote:
      socket.makefile () may lose data when "connection reset by peer".
      and socket.recv() will never lose the data.
      >
      change the "1" to "0" in the client code to see the difference.
      >
      confirmed on both windows and linux.
      >
      so I guess there is a problem with makefile().
      >
      # Echo server program
      import socket
      >
      HOST = '' # Symbolic name meaning the local host
      PORT = 50007 # Arbitrary non-privileged port
      s = socket.socket(s ocket.AF_INET, socket.SOCK_STR EAM)
      s.setsockopt(so cket.SOL_SOCKET , socket.SO_REUSE ADDR, 1)
      s.bind((HOST, PORT))
      s.listen(1)
      while(1):
      conn, addr = s.accept()
      print 'Connected by', addr
      conn.settimeout (1)
      toread = 99
      retrytime = 0
      reads = 0
      while reads < toread and retrytime < 10:
      try:
      data = conn.recv(min(3 2,toread-reads))
      if not data: continue
      print data
      reads += len(data)
      except:
      retrytime += 1
      print "timeout %d" % retrytime
      continue
      #conn.shutdown( socket.SHUT_RD) #no more read
      if reads == toread:
      conn.send("OK")
      else:
      conn.send("NOT OK")
      conn.close()
      >
      # Echo client program
      import socket
      >
      HOST = 'localhost' # The remote host
      PORT = 50007 # The same port as used by the server
      s = socket.socket(s ocket.AF_INET, socket.SOCK_STR EAM)
      s.connect((HOST , PORT))
      for i in range(12):
      print "time %d" % i
      try:
      s.send('0123456 789')
      except socket.error, e:
      print "socket error:", e
      break
      #data = s.recv(1024)
      #print "data %d" %i, data
      #s.shutdown(soc ket.SHUT_WR)#no more write
      >
      '''
      try changing 1 to 0.
      '''
      if 1:
      data=s.recv(102 4)
      else:
      rf = s.makefile("rb" )
      data = rf.read()
      rf.close()
      s.close()
      print 'Received', repr(data)
      >
      The big problem here seems to be that your server is closing the socket
      after reading 99 bytes, but the client is actually sending 120. If you
      change the "99" in the server to "120" you will see that the client
      works when it uses the makefile().read (). I can't be bothered to debug
      the exact reason why the 21 bytes of buffered data doesn't cause a
      problem with the recv() version, but I wouldn't regard the error you are
      getting as a bug in Python, rather as a bug in your application.
      >
      The underlying cause of all this appears to be your failure to
      understand that network protocols should ideally allow the endpoints to
      determine when a complete message has been transmitted, and to consume
      all transmitted data.
      >
      You can't just ignore some or all of a client request and expect it not
      to cause problems. If you look at some of the better-known TCP-based
      application protocols you will see they take pains to ensure that
      clients and servers can deterministical ly parse each others' messages.
      >
      In summary, a better protocol design will cause you rather less grief
      and a little more humility will result in less acidity from crabby old
      geeks like me.
      >
      regards
      Steve
      --
      Steve Holden +1 571 484 6266 +1 800 494 3119
      Holden Web LLC/Ltd http://www.holdenweb.com
      Skype: holdenweb http://del.icio.us/steve.holden
      --------------- Asciimercial ------------------
      Get on the web: Blog, lens and tag the Internet
      Many services currently offer free registration
      ----------- Thank You for Reading -------------- Hide quoted text -
      >
      - Show quoted text -
      I don't think you're crabby, Steve.

      -- Paul

      Comment

      • ahlongxp

        #4
        Re: socket.makefile () buggy?

        On Jul 8, 9:54 pm, Steve Holden <s...@holdenweb .comwrote:
        That's a pretty pejorative subject line for someone who's been
        programming Python [guessing by the date of your first post] for about a
        month.
        >
        I have to admit it that I'm quite a newbie programmer.
        Perhaps "Incomprehensib le behavior from socket.makefile ()", or "I have
        written a buggy network application"? That would at least show that you
        are considering the possibility you yourself are the cause of the
        problem ;-)
        >
        Thanks. I'll be more careful about my words.
        but I did show some possibility myself is the cause of the problem.
        Is there any chance you missed the "?" and "guess" at the same time.
        Python has been around for a long time, so you should ask yourself how
        likely it is that you would be the first to discover such a fundamental
        flaw?
        very low but not zero.
        Miracle happens.
        >I'd be very surprised if someone doesn't point you at an article
        on "how to ask smart questions", but I will content myself with that
        oblique reference.
        >
        >
        The big problem is that, I didn't know such a person like you before.
        It's my misfortune.

        The big problem here seems to be that your server is closing the socket
        after reading 99 bytes, but the client is actually sending 120. If you
        change the "99" in the server to "120" you will see that the client
        works when it uses the makefile().read (). I can't be bothered to debug
        the exact reason why the 21 bytes of buffered data doesn't cause a
        problem with the recv() version, but I wouldn't regard the error you are
        getting as a bug in Python, rather as a bug in your application.
        >
        I don't know the underlying details of networking.
        But with recv() version I can get things done as expected while
        makefile()
        version can't.
        I guess it's not that unreasonable to make a guess like I did.
        The underlying cause of all this appears to be your failure to
        understand that network protocols should ideally allow the endpoints to
        determine when a complete message has been transmitted, and to consume
        all transmitted data.
        >
        You can't just ignore some or all of a client request and expect it not
        to cause problems.
        Back to the problem, I make it working by adding a sleep(0.5) just
        before the
        server closes the connection.
        Actually this is Hendrik van Rooyen's idea.
        And luckily I got a another lesson

        >If you look at some of the better-known TCP-based
        application protocols you will see they take pains to ensure that
        clients and servers can deterministical ly parse each others' messages.
        >
        I found something useful in RFC2616--Hypertext Transfer Protocol --
        HTTP/1.1.
        [PAGE49]
        - If an origin server receives a request that does not include
        an
        Expect request-header field with the "100-continue"
        expectation,
        the request includes a request body, and the server responds
        with a final status code before reading the entire request
        body
        from the transport connection, then the server SHOULD NOT
        close
        the transport connection until it has read the entire request,
        or until the client closes the connection. Otherwise, the
        client
        might not reliably receive the response message. However, this
        requirement is not be construed as preventing a server from
        defending itself against denial-of-service attacks, or from
        badly broken client implementations .

        "Otherwise, the client might not reliably receive the response
        message".

        My problem seems fixed. But it's not "relialbe" though it's working
        pretty
        good under windows 2k, windowsxp, and linux.
        I may reconsider my design and willprobably try dividing request with
        body into two steps like HTTP does.
        In summary, a better protocol design will cause you rather less grief
        and a little more humility will result in less acidity from crabby old
        geeks like me.
        >
        I do appreciate your response.
        I mean it.
        regards
        Steve
        --
        Steve Holden +1 571 484 6266 +1 800 494 3119
        Holden Web LLC/Ltd http://www.holdenweb.com
        Skype: holdenweb http://del.icio.us/steve.holden
        --------------- Asciimercial ------------------
        Get on the web: Blog, lens and tag the Internet
        Many services currently offer free registration
        ----------- Thank You for Reading -------------

        And last but not least, I' here to be helped and help as long as I
        can.
        But as you noticed I'm not very good at expressing myself in English.
        I didn't mean to offense anyone but I might seem to be rude or
        offensive.
        I hope you can understand.


        --
        ahlongxp

        Software College,Northea stern University,Chin a
        ahlongxp@gmail. com


        Comment

        • Steve Holden

          #5
          Re: socket.makefile () buggy?

          ahlongxp wrote:
          On Jul 8, 9:54 pm, Steve Holden <s...@holdenweb .comwrote:
          >That's a pretty pejorative subject line for someone who's been
          >programming Python [guessing by the date of your first post] for about a
          >month.
          >>
          [...]
          And last but not least, I' here to be helped and help as long as I
          can.
          But as you noticed I'm not very good at expressing myself in English.
          I didn't mean to offense anyone but I might seem to be rude or
          offensive.
          I hope you can understand.
          >
          Sure. I felt it necessary to explain my remarks as my own remarks seemed
          potentially rude or offensive. Naturally not everyone has English as a
          first language, but your first post was good enough that it wasn't
          immediately obvious in your case.

          regards
          Steve
          --
          Steve Holden +1 571 484 6266 +1 800 494 3119
          Holden Web LLC/Ltd http://www.holdenweb.com
          Skype: holdenweb http://del.icio.us/steve.holden
          --------------- Asciimercial ------------------
          Get on the web: Blog, lens and tag the Internet
          Many services currently offer free registration
          ----------- Thank You for Reading -------------

          Comment

          • John J. Lee

            #6
            Re: socket.makefile () buggy?

            Steve Holden <steve@holdenwe b.comwrites:
            ahlongxp wrote:
            >On Jul 8, 9:54 pm, Steve Holden <s...@holdenweb .comwrote:
            >>That's a pretty pejorative subject line for someone who's been
            >>programming Python [guessing by the date of your first post] for about a
            >>month.
            >>>
            [...]
            >And last but not least, I' here to be helped and help as long as I
            >can.
            >But as you noticed I'm not very good at expressing myself in English.
            >I didn't mean to offense anyone but I might seem to be rude or
            >offensive.
            >I hope you can understand.
            >>
            Sure. I felt it necessary to explain my remarks as my own remarks
            seemed potentially rude or offensive. Naturally not everyone has
            English as a first language, but your first post was good enough that
            it wasn't immediately obvious in your case.
            This is the danger in getting too good at another language :-)
            ahlongxp, maybe you should drop some deliberate mistakes in your
            too-good English ;-)

            And FWLIW, I'd be less surprised than Steve at your finding a real bug
            in Python's socket code...


            John

            Comment

            • ahlongxp

              #7
              Re: socket.makefile () buggy?

              On Jul 11, 7:51 am, j...@pobox.com (John J. Lee) wrote:
              Steve Holden <s...@holdenweb .comwrites:
              ahlongxp wrote:
              On Jul 8, 9:54 pm, Steve Holden <s...@holdenweb .comwrote:
              >That's a pretty pejorative subject line for someone who's been
              >programming Python [guessing by the date of your first post] for about a
              >month.
              >
              [...]
              And last but not least, I' here to be helped and help as long as I
              can.
              But as you noticed I'm not very good at expressing myself in English.
              I didn't mean to offense anyone but I might seem to be rude or
              offensive.
              I hope you can understand.
              >
              Sure. I felt it necessary to explain my remarks as my own remarks
              seemed potentially rude or offensive. Naturally not everyone has
              English as a first language, but your first post was good enough that
              it wasn't immediately obvious in your case.
              >
              This is the danger in getting too good at another language :-)
              ahlongxp, maybe you should drop some deliberate mistakes in your
              too-good English ;-)
              >
              And FWLIW, I'd be less surprised than Steve at your finding a real bug
              in Python's socket code...
              >
              John
              Sorry, I don't think I understand what you are trying to say.
              But thanks for your response anyway.

              Frank Swarbrick, thank your for your advice. I'll think about it.

              --
              ahlongxp

              Software College,Northea stern University,Chin a
              ahlongxp@gmail. com


              Comment

              Working...