Problems Returning an HTTP 200 Ok Message

Collapse
This topic is closed.
X
X
 
  • Time
  • Show
Clear All
new posts
  • Guy Davidson

    Problems Returning an HTTP 200 Ok Message

    Hi Folks,

    I'm having some issues with an small socket based server I'm writing,
    and I was hoping I could get some help.

    My code (attached below) us supposed to read an HTTP Post message
    coming from a power meter, parse it, and return a proper HTTP 200 Ok
    message. The problem is that the socket fails to send the entire
    message as one message, creating a fragmented message which the power
    meter then fails to read and accept.

    Is there any way to force the socket to send the entire message at
    once? Am I doing anything wrong? Is there an easier way to implement
    this functionality?

    Thanks,

    Guy Davidson

    Code:

    # server application to read data from power meter
    # listens on port 80
    # receives HTTP POST messages

    # author: Maria Kazandjieva <mariakaz@cs.st anford.edu>, Guy Davidson
    <gudyav2002@yah oo.com>
    # updated: July 7th, 2008: staring to add database writing
    functionality

    import time
    import datetime #datetime for response message
    from socket import * #socket module
    import DataPacket #local file with data packet class
    import library
    import MySQLdb

    #formats a proper response message

    myHost = ""
    myPort = 80

    #response = "HTTP/1.1 200 OK\r\nDate: Fri, 20 June 2008 20:40:34 GMT\r
    \nServer:SING\r \nX-Powered-By: maria\r\nConten t-Length: 7\r
    \nConnection:cl ose\r\nContent-Type: text/html\r\n\r\n[0!:20]\n"

    #initializing the socket:
    s = socket(AF_INET, SOCK_STREAM) # create a TCP socket
    s.setsockopt(SO L_SOCKET, SO_REUSEADDR, 1) # allow port reuse? I hope
    s.bind((myHost, myPort)) # bind it to the server port
    s.listen(50) # allow 50 simultaneous
    pending connections
    print('Socket initialized')

    #intializing the database
    db = MySQLdb.connect (host='localhos t', user='powernet' ,
    passwd='jplicks ', db='test')
    cursor = db.cursor()
    print('Database cursor intialized')

    while 1:
    print('In while 1')
    connection, address = s.accept() # connection is a new socket
    print('Accepted a connection')

    # TODO:
    # verify connection is from powernet so we don't get random
    connections to port 80

    # best way to get the data would be:
    # read until we get all the headers
    # parse Content header to get length of data
    # read for that many additional bytes

    while 1:
    print('In while 1 2')
    data = connection.recv (4096) # receive up to 4K bytes
    print('First set of data receieved:')
    #print(data)
    '''
    if data:
    print data
    #connection.sen d(response) #send response; OK 200 HTTP message
    else:
    break
    '''
    if data:
    # second read should recieve the rest of the packet
    # finding how long the content is
    length = data.find('Cont ent-Length: ')
    print('Index of "Content-Length" = ' + str(length))
    length = int(data[length+16:lengt h+18])
    # adding the bytes from the "Content-Type section:"
    length += 51

    data += connection.recv (length)
    print('Second set of data received:')
    print(data)

    if data:
    #print('Found data:')
    #print data #print the string data
    packet = DataPacket.Data Packet(data) #use the string data to
    create a DataPacket object
    print(packet) #print the contents of the object, I hope :)

    #we have a packet, let's write it to the mysql DB
    packet.mysql_in sert(cursor)

    #connection.sen d(response) #send response; OK 200 HTTP
    message
    print('Sending response')
    response = library.respons e_message() #get the response message
    totalsent = 0 #initialize a variable to track how much we've sent
    so far
    while totalsent < len(response): #while we haven't sent everything
    sent = connection.send (response[totalsent:]) #send whatever we
    can starting where we stopped
    totalsent += sent #increment the count of how much we've sent.

    else:
    print('No data')
    break

    else:
    print('No data')
    break
  • Joshua Kugler

    #2
    Re: Problems Returning an HTTP 200 Ok Message

    Guy Davidson wrote:
    Hi Folks,
    >
    I'm having some issues with an small socket based server I'm writing,
    and I was hoping I could get some help.
    >
    My code (attached below) us supposed to read an HTTP Post message
    coming from a power meter, parse it, and return a proper HTTP 200 Ok
    message. The problem is that the socket fails to send the entire
    message as one message, creating a fragmented message which the power
    meter then fails to read and accept.
    >
    Is there any way to force the socket to send the entire message at
    once? Am I doing anything wrong? Is there an easier way to implement
    this functionality?
    I don't have a solution to the problem presented, per se, but you might have
    an easier time implementing this if you use SimpleHTTPServe r or
    CGIHTTPServer modules. This will take care of all the socket listening,
    and a lot of other low-level details.

    Hope that helps!

    j

    Comment

    • samwyse

      #3
      Re: Problems Returning an HTTP 200 Ok Message

      On Jul 10, 1:50 pm, Guy Davidson <GDavids...@gma il.comwrote:
      Hi Folks,
      >
      I'm having some issues with an small socket based server I'm writing,
      and I was hoping I could get some help.
      >
      My code (attached below) us supposed to read an HTTP Post message
      coming from a power meter, parse it, and return a proper HTTP 200 Ok
      message. The problem is that the socket fails to send the entire
      message as one message, creating a fragmented message which the power
      meter then fails to read and accept.
      >
      Is there any way to force the socket to send the entire message at
      once? Am I doing anything wrong? Is there an easier way to implement
      this functionality?
      By 'message', do you mean a single IP datagram? In general, the
      answer is no. Each call to 'connection.sen d()' will (in general, see
      the next paragraph) transmit as much data as will fit into a single IP
      datagram, given the current MTU for the transmission circuit. The
      fact that you're calling it in a loop indicates that the data being
      sent may be larger than will fit into a datagram.

      Or, by 'message', do you mean a single TCP segment? Again, the answer
      is no. Your network stack will try to make the TCP segments the right
      size to fit within a single IP datagram, leading to the same result as
      above.

      From your description, I get the feeling that your power meter has a
      broken network stack, and you're trying to program around it. You
      need to repair the meter.

      Comment

      • Guy Davidson

        #4
        Re: Problems Returning an HTTP 200 Ok Message

        On Jul 10, 12:38 pm, samwyse <samw...@gmail. comwrote:
        On Jul 10, 1:50 pm, Guy Davidson <GDavids...@gma il.comwrote:
        >
        Hi Folks,
        >
        I'm having some issues with an small socket based server I'm writing,
        and I was hoping I could get some help.
        >
        My code (attached below) us supposed to read an HTTP Post message
        coming from a power meter, parse it, and return a proper HTTP 200 Ok
        message. The problem is that the socket fails to send the entire
        message as one message, creating a fragmented message which the power
        meter then fails to read and accept.
        >
        Is there any way to force the socket to send the entire message at
        once? Am I doing anything wrong? Is there an easier way to implement
        this functionality?
        >
        By 'message', do you mean a single IP datagram?  In general, the
        answer is no.  Each call to 'connection.sen d()' will (in general, see
        the next paragraph) transmit as much data as will fit into a single IP
        datagram, given the current MTU for the transmission circuit.  The
        fact that you're calling it in a loop indicates that the data being
        sent may be larger than will fit into a datagram.
        >
        Or, by 'message', do you mean a single TCP segment?  Again, the answer
        is no.  Your network stack will try to make the TCP segments the right
        size to fit within a single IP datagram, leading to the same result as
        above.
        >
        From your description, I get the feeling that your power meter has a
        broken network stack, and you're trying to program around it.  You
        need to repair the meter.
        Here's the weird thing. I know it should be able to be done.

        Let me try and explain in some more depth what I'm trying to do:

        The meter sends HTTP Post messages, and expects, as a reply, an HTTP
        200/Ok message.

        I try to send the following message, using the socket.send() command:

        'HTTP/1.1 200 OK\r\nDate: Thu, 10 July 2008 14:07:50 GMT\r\nServer:
        Apache/2.2.8 (Fedora)\r\nX-Powered-By: PHP/5.2.4\r\nConten t-Length: 4\r
        \nConnection: close\r\nConten t-Type: text/html; charset=UTF-8\r\n\r
        \n[0]\n'

        However, when I snoop on the packets in wireshark, here's what I see:

        HTTP/1.1 200 Ok:

        HTTP/1.1 200 OK
        Date: Wed, 09 July 2008 14:55:50 GMT
        Server: Apache/2.2.8 (Fedora)
        X-Powered-By:

        Continuation or non-HTTP traffic:

        PHP/5.2.4
        Content-Length: 4
        Connection: close
        Content-Type: text/html; charset=UTF-8

        [0]

        It splits into two packages, which the meter can't read, and the
        communication breaks down there.

        Any ideas?

        Thanks.

        Comment

        • Gabriel Genellina

          #5
          Re: Problems Returning an HTTP 200 Ok Message

          En Thu, 10 Jul 2008 18:10:46 -0300, Guy Davidson <GDavidson7@gma il.com>
          escribi�:
          On Jul 10, 12:38 pm, samwyse <samw...@gmail. comwrote:
          >On Jul 10, 1:50 pm, Guy Davidson <GDavids...@gma il.comwrote:
          >>
          My code (attached below) us supposed to read an HTTP Post message
          coming from a power meter, parse it, and return a proper HTTP 200 Ok
          message. The problem is that the socket fails to send the entire
          message as one message, creating a fragmented message which the power
          meter then fails to read and accept.
          >>
          >From your description, I get the feeling that your power meter has a
          >broken network stack, and you're trying to program around it.  You
          >need to repair the meter.
          >
          The meter sends HTTP Post messages, and expects, as a reply, an HTTP
          200/Ok message.
          >
          I try to send the following message, using the socket.send() command:
          >
          'HTTP/1.1 200 OK\r\nDate: Thu, 10 July 2008 14:07:50 GMT\r\nServer:
          Apache/2.2.8 (Fedora)\r\nX-Powered-By: PHP/5.2.4\r\nConten t-Length: 4\r
          \nConnection: close\r\nConten t-Type: text/html; charset=UTF-8\r\n\r
          \n[0]\n'
          >
          However, when I snoop on the packets in wireshark, here's what I see:
          >
          HTTP/1.1 200 Ok:
          >
          HTTP/1.1 200 OK
          Date: Wed, 09 July 2008 14:55:50 GMT
          Server: Apache/2.2.8 (Fedora)
          X-Powered-By:
          >
          Continuation or non-HTTP traffic:
          >
          PHP/5.2.4
          Content-Length: 4
          Connection: close
          Content-Type: text/html; charset=UTF-8
          >
          [0]
          >
          It splits into two packages, which the meter can't read, and the
          communication breaks down there.
          >
          Any ideas?
          As Guy Davidson has already pointed out, this is a problem in the meter
          TCP implementation, and you should ask the vendor to fix it. (Anyway,
          looks like "somewhere" there is a buffer size of 100 bytes or so, very
          small).

          As a workaround, try to shorten your HTTP response; I guess the Server and
          X-Powered-By headers are not required; the Date probably isn't either; and
          if all your responses are like "[0]" a simple "Content-Type: text/plain"
          may suffice (the meter might just ignore it, anyway).

          --
          Gabriel Genellina

          Comment

          • samwyse

            #6
            Re: Problems Returning an HTTP 200 Ok Message

            On Jul 10, 4:10 pm, Guy Davidson <GDavids...@gma il.comwrote:
            I try to send the following message, using the socket.send() command:
            >
            'HTTP/1.1 200 OK\r\nDate: Thu, 10 July 2008 14:07:50 GMT\r\nServer:
            Apache/2.2.8 (Fedora)\r\nX-Powered-By: PHP/5.2.4\r\nConten t-Length: 4\r
            \nConnection: close\r\nConten t-Type: text/html; charset=UTF-8\r\n\r
            \n[0]\n'
            >
            However, when I snoop on the packets in wireshark, here's what I see:
            >
            HTTP/1.1 200 Ok:
            >
            HTTP/1.1 200 OK
            Date: Wed, 09 July 2008 14:55:50 GMT
            Server: Apache/2.2.8 (Fedora)
            X-Powered-By:
            >
            Continuation or non-HTTP traffic:
            >
            PHP/5.2.4
            Content-Length: 4
            Connection: close
            Content-Type: text/html; charset=UTF-8
            >
            [0]
            >
            It splits into two packages, which the meter can't read, and the
            communication breaks down there.
            OK, it looks like a single TCP segment is being sent by you (which is
            consistent with only one socket.send() command being needed), but
            something along the way to the meter is using an MTU (Maximum
            Transmission Unit) of about 100 bytes, producing two IP datagrams.
            How many hops are there between the PC and the meter? Are you
            sniffing on the same subnet as the PC, the meter, or somewhere in
            between? MTU is normally set to about 1500 (and MSS is generally
            MTU-40), but you can generally change these values.

            You should be able to set the do-not-fragment flag on your IP packets,
            but that may cause them to get dropped instead of sent. You could
            also try setting a smaller ICP MSS (Maximum Segment Size), which the
            meter might be able to assemble, even if it can't handle fragmented IP
            datagrams. Try http://www.cisco.com/en/US/tech/tk87...html#prob_desc
            for more help.

            You really need to get a networking guru involved if you want to go
            down this path. I used to be one, but that was years ago. Or, you
            can take Gabriel Genellina's advice and try coding smaller responses.

            Comment

            • samwyse

              #7
              Re: Problems Returning an HTTP 200 Ok Message

              On Jul 11, 3:46 am, "Gabriel Genellina" <gagsl-...@yahoo.com.a r>
              wrote:
              As Guy Davidson has already pointed out, this is a problem in the meter  
              TCP implementation, and you should ask the vendor to fix it.
              That would have been me, not Guy.

              Comment

              • Gabriel Genellina

                #8
                Re: Problems Returning an HTTP 200 Ok Message

                En Mon, 14 Jul 2008 17:46:12 -0300, samwyse <samwyse@gmail. comescribió:
                On Jul 11, 3:46 am, "Gabriel Genellina" <gagsl-...@yahoo.com.a r>
                wrote:
                >
                >As Guy Davidson has already pointed out, this is a problem in the meter
                That would have been me, not Guy.
                Indeed, sorry the misattribution!

                --
                Gabriel Genellina

                Comment

                Working...