reading into the buffer

[puzzlecracker]

Say I want to arrange bytes in the internal buffer in a certain way. I
receive those bytes in the socket.

One solution is to read in socket in pieces:

byte[] buffer = new byte[65536];
int index = 0;
m_socket.Receiv e(buffer , 8, SocketFlags.Non e);
index += 8;
//read the rest of the message
//int msgLength is remaining number of bytes in the buffer
m_socket.Receiv e(buffer , index, msgLen - index,
SocketFlags.Non e);


Questions:

1. How to calculate msgLength ?
2. How to make this operation when reading data in asynchronously,
using BeginReceive()

Thanks

[Peter Duniho]

Re: reading into the buffer

On Fri, 24 Oct 2008 10:08:42 -0700, puzzlecracker <ironsel2000@gm ail.com>
wrote:

Say I want to arrange bytes in the internal buffer in a certain way. I
receive those bytes in the socket.
>
One solution is to read in socket in pieces:
>
byte[] buffer = new byte[65536];
int index = 0;
m_socket.Receiv e(buffer , 8, SocketFlags.Non e);
index += 8;
//read the rest of the message
//int msgLength is remaining number of bytes in the buffer
m_socket.Receiv e(buffer , index, msgLen - index,
SocketFlags.Non e);
>
>
Questions:
>
1. How to calculate msgLength ?

If by "remaining number of bytes in the buffer", you mean ready to be read
from the socket, the answer is "you don't". For the same reason that
using a method like Poll() or Select() cannot provide any guarantees about
what will happen when you actually try to read from the socket, there's no
reliable way to calculate the bytes yet unread in the socket buffer in a
way that will allow you to depend on the calculation in a later call to
Receive().

The best you can do is provide a buffer to Receive() that is just large
enough for the remaining bytes you expect to get for a given message (I'm
assuming that the data in the first eight bytes in some way allow you to
calculate that). Then just keep reading in a loop until you've read that
many bytes, decreasing the length you offer to Receive() with each
iteration to take into account the number of bytes read already.

That said, this is only something you should be doing if you don't need
efficient i/o. There's a lot of overhead going back and forth between
your code and the network buffers, and if there's a lot of data coming
through, you can wind up forcing the network driver to have to throw out
incoming data, and making the other end send it again.

A much better approach is for the network i/o code to always just read as
much data as it can, and let another layer of your code deal with parsing
that out into usable messages. Doing so won't actually necessarily
address the above question directly (after all, the network i/o layer is
probably still going to be delivering data as a stream), but it does shift
it into a domain where you have a bit more control.

2. How to make this operation when reading data in asynchronously,
using BeginReceive()

The same way you'd deal with any transition in design from synchronous to
asynchronous. State that is kept in local variables simply has to be
moved into instance variables that can be accessed as each network
operation completes.

Pete

[puzzlecracker]

Re: reading into the buffer

On Oct 24, 1:36 pm, "Peter Duniho" <NpOeStPe...@nn owslpianmk.com>
wrote:

On Fri, 24 Oct 2008 10:08:42 -0700, puzzlecracker <ironsel2...@gm ail.com> 
wrote:
>
>
>

Say I want to arrange bytes in the internal buffer in a certain way. I
receive those bytes in the socket.

>

One solution is to read in socket in pieces:

>

            byte[] buffer = new byte[65536];
            int index = 0;
            m_socket.Receiv e(buffer , 8, SocketFlags.Non e);
             index += 8;
            //read the rest of the message
            //int msgLength is remaining number of bytes inthe buffer
            m_socket.Receiv e(buffer , index, msgLen - index,
SocketFlags.Non e);

>

Questions:

>

1. How to calculate  msgLength ?

>
If by "remaining number of bytes in the buffer", you mean ready to be read  
 from the socket, the answer is "you don't".  For the same reason that  
using a method like Poll() or Select() cannot provide any guarantees about  
what will happen when you actually try to read from the socket, there's no  
reliable way to calculate the bytes yet unread in the socket buffer in a  
way that will allow you to depend on the calculation in a later call to  
Receive().
>
The best you can do is provide a buffer to Receive() that is just large  
enough for the remaining bytes you expect to get for a given message (I'm 
assuming that the data in the first eight bytes in some way allow you to  
calculate that).  Then just keep reading in a loop until you've read that  
many bytes, decreasing the length you offer to Receive() with each  
iteration to take into account the number of bytes read already.
>
That said, this is only something you should be doing if you don't need  
efficient i/o.  There's a lot of overhead going back and forth between  
your code and the network buffers, and if there's a lot of data coming  
through, you can wind up forcing the network driver to have to throw out  
incoming data, and making the other end send it again.
>
A much better approach is for the network i/o code to always just read as 
much data as it can, and let another layer of your code deal with parsing 
that out into usable messages.  Doing so won't actually necessarily  
address the above question directly (after all, the network i/o layer is  
probably still going to be delivering data as a stream), but it does shift  
it into a domain where you have a bit more control.
>

2. How to make this operation when reading data in asynchronously,
using BeginReceive()

>
The same way you'd deal with any transition in design from synchronous to 
asynchronous.  State that is kept in local variables simply has to be  
moved into instance variables that can be accessed as each network  
operation completes.
>
Pete

Is there another .net group dedicated to networking issues related to
csharp and .net general?

[Peter Duniho]

Re: reading into the buffer

On Fri, 24 Oct 2008 13:07:02 -0700, puzzlecracker <ironsel2000@gm ail.com>
wrote:

Is there another .net group dedicated to networking issues related to
csharp and .net general?

There are no networking issues related to C#. The API is
language-agnostic.

As far as a newsgroup specific to network code in .NET, none that I'm
aware of. However, note that -- as I've mentioned before -- to a large
extent these issues aren't even specific to .NET; they are either inherent
in networking generally, or in Winsock, on which the .NET networking API
is built.

If you want a newsgroup that is more specific to networking questions than
this one, you might try alt.winsock.pro gramming or
comp.os.ms-windows.program mer.tools.winso ck. There are other newsgroups
that are even more general than those, not even being specific to Winsock.

Pete

[Steve]

Re: reading into the buffer

"puzzlecrac ker" <ironsel2000@gm ail.comwrote in message
news:528e671a-3aa2-422c-885c-5e5c50be8334@f7 7g2000hsf.googl egroups.com...

Say I want to arrange bytes in the internal buffer in a certain way. I
receive those bytes in the socket.
>
One solution is to read in socket in pieces:
>
byte[] buffer = new byte[65536];
int index = 0;
m_socket.Receiv e(buffer , 8, SocketFlags.Non e);
index += 8;
//read the rest of the message
//int msgLength is remaining number of bytes in the buffer
m_socket.Receiv e(buffer , index, msgLen - index,
SocketFlags.Non e);
>
>
Questions:
>
1. How to calculate msgLength ?
2. How to make this operation when reading data in asynchronously,
using BeginReceive()
>
Thanks

The receive funciton returns the number of bytes received. The actual
number received may be (and often is) less than the number of bytes
requested (what I consider to be a quirk of BSD sockets). Here is a small
example of some working code that receives the next 4 bytes from a socket
stream.

Byte[] headerBuffer = new Byte[ 4 ];
int bytesReceived = 0;
int bytesToReceive = 4;
while( bytesToReceive bytesReceived )
{
bytesReceived += clientSocket.Re ceive( headerBuffer,
bytesReceived,
bytesToReceive -
bytesReceived, SocketFlags.Non e );
}

The basic idea is to repeast receiving until the number of byte asked for
are received. The reeive function includes an offset into the buffer to
append received data to, which is conventient so you can just keep sending
the same base buffer address.

I can't help with BeginReceive. I've never used it. I usually run separate
send and receive threads. The send thread takes messages from a queue and
sends them out the socket, and the receive thread receives messages from a
socket and makes the available to a queue that is read from the rest of the
application.

Regards,
Steve

[Peter Duniho]

Re: reading into the buffer

On Sun, 26 Oct 2008 12:45:26 -0700, Steve <nospam_steved9 4@comcast.net>
wrote:

The receive funciton returns the number of bytes received. The actual
number received may be (and often is) less than the number of bytes
requested (what I consider to be a quirk of BSD sockets).

If by "quirk" you mean "essential part of the design of TCP", then yes. :)

The behavior isn't specific to BSD sockets (the API from which Winsock is
inherited). It's a fundamental part of TCP. TCP does not require that
the receiver be able to know how many bytes will be sent in a stream. In
many protocols, the sender simply sends bytes until it's done, and then
shuts down the connection. The receiver knows it's received the last byte
when the shutdown is indicated at the receiver's end (e.g. Receive()
returns 0 as the number of bytes read).

If Receive() always blocked until it'd filled the buffer passed to it,
then applications that use the end of the stream to know when the
transmission has been completed simply wouldn't work.

Even changing the API so that it only returned with a partially-filled
buffer at the end-of-stream wouldn't make sense, because another valid way
of terminating messages is with a delimiter (for example, sending
null-terminated strings). Again, the receiver doesn't have advance
knowledge of the number of bytes that will be sent, so requiring to
provide that information in the form of the buffer length wouldn't work.

Instead, with all of the socket-based APIs, the receiving function
(recv(), WSARecv(), Socket.Receive( ), NetworkStream.R ead(), etc.) will
always return as much data is _available_ at the time that the operation
can be completed, up to the size of the buffer passed in. It never waits
to try to fill the buffer, because that wait could be indeterminately
long, or even infinitely long.

Here is a small
example of some working code that receives the next 4 bytes from a socket
stream. [...]

Not counting the poor efficiency, code like that is fine if you know for
sure that you're going to get 4 bytes, and you know for sure that you
can't do anything useful with the data until you've received those 4 bytes.

But that's hardly a universal condition for network code. It's not
something one would want generally built into the network API itself, and
I don't really think that "quirk" is the right word to describe how TCP
and the APIs available to use it work.

Pete

[Steve]

Re: reading into the buffer

"Peter Duniho" <NpOeStPeAdM@nn owslpianmk.comw rote in message
news:op.ujnjhbb k8jd0ej@petes-computer.local. ..

On Sun, 26 Oct 2008 12:45:26 -0700, Steve <nospam_steved9 4@comcast.net>
wrote:
>

[snip]

>
But that's hardly a universal condition for network code. It's not
something one would want generally built into the network API itself, and
I don't really think that "quirk" is the right word to describe how TCP
and the APIs available to use it work.
>
Pete

The behavior you describe (and the way the receive works) is what I would
have expected from non-blocking sockets. With non-blocking sockets I would
expect to use "select" if I want to wait for data to show up, which perits a
timeout, and then use recv to get whatever data is available.

With blocking sockets I would expect that if I say I want to receive 128
characters, then I know I am expecting 128 characters, and don't bother me
until they show up. I know that isn't the way it works, but that was the
behavior I was expecting before I learned how things really work.

In fact I think the API would be easier to use if I could say receive n
bytes of data, or timeout if no new data shows up for u milliseconds or the
entire message doesn't show up in v milliseconds.

But, we live with what we have... and it works.

Regards,
Steve

[Peter Duniho]

Re: reading into the buffer

On Sun, 26 Oct 2008 18:47:41 -0700, Steve <nospam_steved9 4@comcast.net>
wrote:

"Peter Duniho" <NpOeStPeAdM@nn owslpianmk.comw rote in message
news:op.ujnncbd w8jd0ej@petes-computer.local. ..
[snip]

>>
>So, it's your assertion that only protocols which advertise the number
>of
>bytes to be received in advance should be allowed to be implemented
>using
>blocking sockets?

>
It would certainly be more intuitive.

It would be _more_ intuitive for the API's buffer-management semantics to
vary according to a completely unrelated aspect of the API? And in a way
that prevents entire classes of protocol implementations from ever being
implemented using one of the two basic i/o mechanisms offered by the API?

If that's what's intuitive to you, all I can say is that I'm so very glad
you're not in the business of designing the APIs I use. Suffice to say,
you are in a very tiny minority in your opinion of how sockets should have
been designed.

[...]
I don't buy the argument that "it wouldn't work at all if designed
without
the behaviour you say is a quirk".

Then disprove it. Show how the API would work if a blocking call to
receive data did not return until the buffer passed to it had filled, even
with protocols where the receiver has no way to know how many bytes to
expect.

If you had to wait on one call that
tells you when and how much data is available, and then do a recv call to
read that much data, it would work just as well as the existing recv.

First of all, semantically that's exactly what happens now, except you can
do it in a single call. Secondly, from a practical point of view, that
API wouldn't work at all, because there's no reliable way to query the
amount of data that's available and be guaranteed that data will _still_
be available when you try to receive it (in case you missed it earlier, I
refer to the Winsock Lame List link I provided previously:
http://tangentsoft.net/wskfaq/articles/lame-list.html).

The entire Internet is founded on some fundamental design choices about
how TCP/IP works, including the option of endpoints to drop received data
and force a retry if they become overloaded or otherwise need to clear
some buffer space. These very basic design choices affect a number of
things, including how an API like sockets _has_ to work.

You can rail against it if you like, but all you're going to do is make
yourself unhappy. The fact is, given the technology involved, there's
really no way that the sockets API could have been designed to work as you
want it to.

Pete

[puzzlecracker]

Re: reading into the buffer

On Oct 24, 5:53 pm, "Peter Duniho" <NpOeStPe...@nn owslpianmk.com>
wrote:

On Fri, 24 Oct 2008 13:07:02 -0700, puzzlecracker <ironsel2...@gm ail.com> 
wrote:
>

Is there another .net group dedicated to networking issues related to
csharp and .net general?

>
There are no networking issues related to C#.  The API is  
language-agnostic.
>
As far as a newsgroup specific to network code in .NET, none that I'm  
aware of.  However, note that -- as I've mentioned before -- to a large 
extent these issues aren't even specific to .NET; they are either inherent  
in networking generally, or in Winsock, on which the .NET networking API  
is built.
>
If you want a newsgroup that is more specific to networking questions than  
this one, you might try alt.winsock.pro gramming or  
comp.os.ms-windows.program mer.tools.winso ck.  There are other newsgroups  
that are even more general than those, not even being specific to Winsock..
>
Pete

Thanks, I'm taking a dive into learning of winsock..

[Steve]

Re: reading into the buffer

"Peter Duniho" <NpOeStPeAdM@nn owslpianmk.comw rote in message
news:op.ujnuheg 18jd0ej@petes-computer.local. ..
[snip]

The entire Internet is founded on some fundamental design choices about
how TCP/IP works, including the option of endpoints to drop received data
and force a retry if they become overloaded or otherwise need to clear
some buffer space. These very basic design choices affect a number of
things, including how an API like sockets _has_ to work.
>

Clearly you have the positon that anything other than the way things are
currently implented will not work.

I'll agree to disagree.

Regards,
Steve

You can rail against it if you like, but all you're going to do is make
yourself unhappy. The fact is, given the technology involved, there's
really no way that the sockets API could have been designed to work as you
want it to.
>
Pete

[Peter Duniho]

Re: reading into the buffer

On Mon, 27 Oct 2008 18:53:51 -0700, Steve <nospam_steved9 4@comcast.net>
wrote:

Clearly you have the positon that anything other than the way things are
currently implented will not work.

Sort of. There are other alternatives. But they will all share the same
basic behavior that you don't like. It's just how TCP/IP works.

I'll agree to disagree.

As I said, you're welcome to. It's not going to help you in any way, but
it's certainly your prerogative.

Pete