TCP Delivery

[Airslash]

Hello,

I hope this is the correct forum to ask this...

I'm working on a client-server architecture application that relies on the TCP/IP protocol for transmitting data between the client and server application.

I've made my own protocol for the application that relies on the following structure:
@VT: <type> <size> [binary block]#

- Each message starts with @VT:
- type is an integer, identifying the type
- size is the length of the binary block
- binary block contains type specific data
- each message ends with #

Currently we're having discussions here whether or not to fully parse incoming data. Since I'm working with TCP, I assume that when I receive data on the incoming socket (C#.NET Sockets), the data contained in the socket is a complete message, meaning it starts with @VT: and ends with #.

Colleagues of mine say I need to parse the buffer everytime to search for an @ symbol and then parse the remaining data.

who is right?

[sicarie]

I'm not sure I completely understand you, but if you've created a protocol within TCP, you should be able to use the tools within TCP to verify if the message is complete or not.

Inside that, you should parse your own protocol for completion anyway - just as a method of error-checking. There are plugins and toolbars that allow you to inject into packets, so input validation is key.

That's just a personal opinion, though - if you have error checking on your client sending info, it's your decision if you want to parse it on the server end.

[Airslash]

What I'm trying to say is that we had a discussion about the TCP itself.

I assume that when TCP reports that data is available on the socket, that I have a complete message available. My colleague says that It is possible that I only have a part of the message on the socket and that I always need to validate the whole content.

I've had occurances of 2 messages beeing in the socket, but havent encountered the issue that I only had a partial message.

[sicarie]

Ah, yes, it's entirely possible - TCP negotiates packet size during the handshake, it is possible the agreed upon size is less than the largest 'packet' from your protocol.

[Banfa]

Sorry to step in late I just want to reinforce what sicarie has said since he is right.

There are 2 normal protocols to use across an enternet, TCP and UDP.

TCP is a connection based byte stream. That is you open a connection and only transmit once you know the connection is open. When you transmit what you transmit is a stream of bytes, that is the tcp stack has no concept of sending messages, it sends data, a minimum of 1 byte and a maximum normally defined by the underlying link layer (a little bit over 1400 bytes on most wire and wireless networks but depends on medium, you often get smaller MTUs on mobile phone networks). TCP also guarantees that if you receive the data it will be correct and in the right order.

The important point here is that it is a byte stream not message orientated, that means the TCP stack may split the data it has anyway it chooses without regard for what you term message boundaries. You might receive bits of a message at a time or several messages all together.

When handling a TCP connection you need to account for this by treating the data as what it is, a byte stream, so your program should be capable of handling data reception a byte at a time and should make no assumptions about what it has received (i.e. having received the message header/size assuming the rest of the data is either available or lost, it just may not have turned up yet).

By contrast UDP is connection less and message based. You do not have to set-up a connection, you transmit the message in a fire and forget manner, that is sending end does not concern itself with worrying about if the data arrived. The delivery is not guaranteed although UDP can (but is not required to) use a checksum in which case you can normally rely on the message being correct if it does arrive. Also UDP makes no guarantee about order of deliver, that is a message sent 2nd may arrive 1st.

Data only arrives if the receiving computer was looking for a message at the time it arrives. Also because UDP sends its data as messages it is desirable where possible to limit the message length to the UDP message so it fits into length of the link layer with the smallest MTU on the route. This prevent fragmentation of the message and that is good because once fragmented only 1 of the fragments needs to be lost to loose the whole message, since there is no re-transmit.

When UDP data arrives it is a whole and single message and the UDP header carries the size of that message. So if you receive a UDP packet you can expect it to be a complete and whole message.

[Airslash]

I finally got around to implementing a nice system around my socket.

Currently it works like this:

- TCP Berkley socket listens for incoming data.
- When data arrives, the socket reads it and write it at the back of a memory buffer.
- A parser uses a background thread to loop over the buffer from the start and analyses the data, seeking the start & end of a message according my protocol
- If the parser finds a message, it removes that data from the buffer and raises an event, then seeks the next message.

Currently working like a charm, still ironing out several bugs about odd situations that can occur, but the general application flow seems to work fine.

Thanks for the explanations. I've taken Sicarie's reply as he/she was first with the explanation (Sorry Banfa :))

[Banfa]

That's OK its not a competition :D

Sounds workable, since you are accessing the buffer from 2 separate threads I assume your using some synchronisation object (a semaphore) to protect the buffer access?

[Airslash]

Yup,

I have written a class called MemoryBuffer that wraps an IList<byte> object and protects it with a lock for exclusive access through every public method.