TcpClient buffer size limit?

[Sharon]

I'm using TcpClient and getting the Stream by:
TcpClient tcpclnt = new TcpClient();
. . .
Stream stm = tcpclnt.GetStre am();

Now, when I'm trying to send a big buffer via stm.Write(...) I get an
exception saying that the buffer is too big.

I tried to set the tcpclnt.Receive BufferSize to the size of the buffer I"m
trying to send, but the exception is still thrown, there is no transmit
buffer I can set.

How can I solve this problem beside splitting my buffer and transmit it by
chunks?
What is the limit of the buffer?

--
Thanks
Sharon G.

[Ignacio Machin \( .NET/ C# MVP \)]

Re: TcpClient buffer size limit?

hi sharon,


weird error, could you post the code section ?

Write do not need to create a new buffer, it;s already created.

cheers,

--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation


"Sharon" <Sharon@discuss ions.microsoft. com> wrote in message
news:0E82306F-E31D-4D7F-B9A4-0BE9D20741C3@mi crosoft.com...[color=blue]
> I'm using TcpClient and getting the Stream by:
> TcpClient tcpclnt = new TcpClient();
> . . .
> Stream stm = tcpclnt.GetStre am();
>
> Now, when I'm trying to send a big buffer via stm.Write(...) I get an
> exception saying that the buffer is too big.
>
> I tried to set the tcpclnt.Receive BufferSize to the size of the buffer I"m
> trying to send, but the exception is still thrown, there is no transmit
> buffer I can set.
>
> How can I solve this problem beside splitting my buffer and transmit it by
> chunks?
> What is the limit of the buffer?
>
> --
> Thanks
> Sharon G.[/color]

[Sharon]

Re: TcpClient buffer size limit?

OK, here is the code:

//////////////////
// The client:
//////////////////
TcpClient tcpclnt = new TcpClient();
tcpclnt.Connect ("ip address", 8001);
byte [] rowData = new byte[104857600];// 100 MByte
tcpclnt.Receive BufferSize = rowData.Length;
Stream stm = tcpclnt.GetStre am();
stm.Write(rowDa ta, 0, rowData.Length) ; // throwing an EXCEPTION !!!
// Receiving Ack.
ack = new byte[100];
k = stm.Read(ack, 0, ack.Length);
tcpclnt.Close() ;

//////////////////
// The Serevr:
//////////////////
ASCIIEncoding encoding = new ASCIIEncoding() ;
IPAddress ipAd = IPAddress.Parse ("ip address");// Same IP address as the
clients uses.
TcpListener myList = new TcpListener(ipA d, 8001);
myList.Start();
Socket sock = myList.AcceptSo cket();
byte [] rowData = new byte[104857600];// 100 MByte
recLen = sock.Receive(ro wData); // throwing an EXCEPTION !!!
sock.Close();
myList.Stop();
///////////////////////////

Any idea ?


--------
Thanks
Sharon

[Ignacio Machin \( .NET/ C# MVP \)]

Re: TcpClient buffer size limit?

hi sharon

I got the same error, this is the first time I see it, how many memory do
you have? I only have 512 and I got reported around 800 when I create both
buffers but before send it, maybe the buffer is too big, I think that you
should do a google search to see if anybody has got this error before


Sorry not be able to help you further :(

cheers,

--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation


"Sharon" <Sharon@discuss ions.microsoft. com> wrote in message
news:79D20349-97FF-40FB-9268-15BA1D4042BD@mi crosoft.com...[color=blue]
> OK, here is the code:
>
> //////////////////
> // The client:
> //////////////////
> TcpClient tcpclnt = new TcpClient();
> tcpclnt.Connect ("ip address", 8001);
> byte [] rowData = new byte[104857600];// 100 MByte
> tcpclnt.Receive BufferSize = rowData.Length;
> Stream stm = tcpclnt.GetStre am();
> stm.Write(rowDa ta, 0, rowData.Length) ; // throwing an EXCEPTION !!!
> // Receiving Ack.
> ack = new byte[100];
> k = stm.Read(ack, 0, ack.Length);
> tcpclnt.Close() ;
>
> //////////////////
> // The Serevr:
> //////////////////
> ASCIIEncoding encoding = new ASCIIEncoding() ;
> IPAddress ipAd = IPAddress.Parse ("ip address");// Same IP address as the
> clients uses.
> TcpListener myList = new TcpListener(ipA d, 8001);
> myList.Start();
> Socket sock = myList.AcceptSo cket();
> byte [] rowData = new byte[104857600];// 100 MByte
> recLen = sock.Receive(ro wData); // throwing an EXCEPTION !!!
> sock.Close();
> myList.Stop();
> ///////////////////////////
>
> Any idea ?
>
>
> --------
> Thanks
> Sharon[/color]

[Sharon]

Re: TcpClient buffer size limit?

Hi,

I also have 512 MByte of memory.
What do you mean when you say: "I got reported around 800 when I create both
buffers but before send it" ??? Only the receive buffer can be set. 800
what? Did you try to set the buffer to 800 MByte or what?

I get the error not when I set the ReceiveBufferSi ze, but when I call stream
Read/Write when my large buffer which is over 100 MByte.

I'll appreciate any clue solving this issue.


-----
Thanks
Sharon

[Helge Jensen]

Re: TcpClient buffer size limit?

Sharon wrote:

Are you *really* sending 100M, or have you just made the buffer
"big-enough"?

Apparently stm.Write invokes the OS on the buffer directly, and the OS
doesn't support a block that big. I Have not seen this before, and
would expect .NET to handle it inside Stream.Write.
[color=blue]
> stm.Write(rowDa ta, 0, rowData.Length) ; // throwing an EXCEPTION !!![/color]

// Warning: untested code
int blockSize = 8192;
int blockCount = rowdata.Length/blockSize
+ rowdata.Length % blockSize == 0 ? 0 : 1;
for ( int i = 0; i < blockCount ; ++i ) {
int length;
if ( i < blockCount - 1 )
length = blockSize;
else
length = rowdata.Length - (i-1)*blockSize;
stm.Write(rowDa ta, i*blockSize, length);
}

The above code could easily have been in .Write instead.
[color=blue]
> // Receiving Ack.
> ack = new byte[100];
> k = stm.Read(ack, 0, ack.Length);[/color]

I assume you just cutted some code checking on the ack for readbility?

BTW: nobody promised you that a read of a 100 byte buffer will return
100 bytes, even if they have arrived. It might return just 1, and then
return the remaining bytes later.
[color=blue]
> byte [] rowData = new byte[104857600];// 100 MByte
> recLen = sock.Receive(ro wData); // throwing an EXCEPTION !!![/color]

Same propblem as above, although I would have probably just limited the
count if I was the implementer of .Receive, afterall Receive is allowed
to return any number of available bytes, or 0 iff the stream is closed.

Here, there is *no* *way* that the OS will return the entire 100MB data
in one read. You need to loop, as when writing.
[color=blue]
> sock.Close();[/color]

Missing ACK? :)


--
Helge Jensen
mailto:helge.je nsen@slog.dk
sip:helge.jense n@slog.dk
-=> Sebastian cover-music: http://ungdomshus.nu <=-

[Ignacio Machin \( .NET/ C# MVP \)]

Re: TcpClient buffer size limit?

Hi,

800 MB I meant, that is what the PF Usage reported ( in windows task manager
, performance tab )

You create both buffer before the transmition ( where you get the error ), I
put a breakpoint there and check the memory usage.

cheers,

--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation


"Sharon" <Sharon@discuss ions.microsoft. com> wrote in message
news:9D241492-52E9-4750-BB27-FEBE3EEA86A5@mi crosoft.com...[color=blue]
> Hi,
>
> I also have 512 MByte of memory.
> What do you mean when you say: "I got reported around 800 when I create
> both
> buffers but before send it" ??? Only the receive buffer can be set. 800
> what? Did you try to set the buffer to 800 MByte or what?
>
> I get the error not when I set the ReceiveBufferSi ze, but when I call
> stream
> Read/Write when my large buffer which is over 100 MByte.
>
> I'll appreciate any clue solving this issue.
>
>
> -----
> Thanks
> Sharon[/color]

[Ignacio Machin \( .NET/ C# MVP \)]

Re: TcpClient buffer size limit?

Hi,

[color=blue]
> Apparently stm.Write invokes the OS on the buffer directly, and the OS
> doesn't support a block that big. I Have not seen this before, and would
> expect .NET to handle it inside Stream.Write.[/color]

That's my same concern , I have never used a buffer this big, but I would
assume that Stream would take care of that for me, hopefully somebody from
MS would clarify this issue



cheers,

--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation

[Willy Denoyette [MVP]]

Re: TcpClient buffer size limit?



"Ignacio Machin ( .NET/ C# MVP )" <ignacio.mach in AT dot.state.fl.us > wrote
in message news:eg4C$PTNFH A.2580@TK2MSFTN GP09.phx.gbl...[color=blue]
> Hi,
>
>[color=green]
>> Apparently stm.Write invokes the OS on the buffer directly, and the OS
>> doesn't support a block that big. I Have not seen this before, and would
>> expect .NET to handle it inside Stream.Write.[/color]
>
> That's my same concern , I have never used a buffer this big, but I would
> assume that Stream would take care of that for me, hopefully somebody
> from MS would clarify this issue
>
>
>
> cheers,
>
> --
> Ignacio Machin,
> ignacio.machin AT dot.state.fl.us
> Florida Department Of Transportation
>
>[/color]
Ignacio,

The error has nothing to do with the framework, the buffer size is limited
by the underlying Winsock protocol stack. The maximum size of the buffer is
determined by the provider using a (undocumented) complex algorithm based on
things like available RAM, socket type, bandwidth number of active
connections.... , a too high size returns WSAENOBUFS (10055) Winsock error
with "An operation on a socket could not be performed because the system
lacked sufficient buffer space or because a queue was full. " as error
message.

I don't understand the reasoning behind OP's decision to reserve 100 MB for
this, anyway 100MB is just way too high.

Willy.

[Ignacio Machin \( .NET/ C# MVP \)]

Re: TcpClient buffer size limit?

Hi,

Yes, I understand where and why the error is showning, ITOH I would have
expect that the framework take this into consideration , by either capturing
the exception and then doing a loop ( what the OP would have to do ) or by
providing a method stating what the maximun buffer can be ( tentatively or
course ).
In the current status it's unpredictible how big the buffer can be.

Of course I do not understand neither why the need of such a HUGE buffer
being used by the OP , but you could scale it back to a PPC with 64MB of ram
and now your constrains are real

cheers,

--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation



"Willy Denoyette [MVP]" <willy.denoyett e@telenet.be> wrote in message
news:uCk$$hWNFH A.3156@TK2MSFTN GP15.phx.gbl...[color=blue]
>
>
> "Ignacio Machin ( .NET/ C# MVP )" <ignacio.mach in AT dot.state.fl.us >
> wrote in message news:eg4C$PTNFH A.2580@TK2MSFTN GP09.phx.gbl...[color=green]
>> Hi,
>>
>>[color=darkred]
>>> Apparently stm.Write invokes the OS on the buffer directly, and the OS
>>> doesn't support a block that big. I Have not seen this before, and
>>> would expect .NET to handle it inside Stream.Write.[/color]
>>
>> That's my same concern , I have never used a buffer this big, but I would
>> assume that Stream would take care of that for me, hopefully somebody
>> from MS would clarify this issue
>>
>>
>>
>> cheers,
>>
>> --
>> Ignacio Machin,
>> ignacio.machin AT dot.state.fl.us
>> Florida Department Of Transportation
>>
>>[/color]
> Ignacio,
>
> The error has nothing to do with the framework, the buffer size is limited
> by the underlying Winsock protocol stack. The maximum size of the buffer
> is determined by the provider using a (undocumented) complex algorithm
> based on things like available RAM, socket type, bandwidth number of
> active connections.... , a too high size returns WSAENOBUFS (10055) Winsock
> error with "An operation on a socket could not be performed because the
> system lacked sufficient buffer space or because a queue was full. " as
> error message.
>
> I don't understand the reasoning behind OP's decision to reserve 100 MB
> for this, anyway 100MB is just way too high.
>
> Willy.
>[/color]

[Willy Denoyette [MVP]]

Re: TcpClient buffer size limit?


"Ignacio Machin ( .NET/ C# MVP )" <ignacio.mach in AT dot.state.fl.us > wrote
in message news:%23MQGKihN FHA.604@TK2MSFT NGP10.phx.gbl.. .[color=blue]
> Hi,
>
> Yes, I understand where and why the error is showning, ITOH I would have
> expect that the framework take this into consideration , by either
> capturing the exception and then doing a loop ( what the OP would have to
> do ) or by providing a method stating what the maximun buffer can be (
> tentatively or course ).
> In the current status it's unpredictible how big the buffer can be.
>
> Of course I do not understand neither why the need of such a HUGE buffer
> being used by the OP , but you could scale it back to a PPC with 64MB of
> ram and now your constrains are real
>
> cheers,
>
> --
> Ignacio Machin,
> ignacio.machin AT dot.state.fl.us
> Florida Department Of Transportation
>[/color]

There is nothing the Framework can do other than you could do, it has no way
to determine the max. buffer size other than trying to send and reducinfg
the size until it succeeds, not exactly what I call a neet solution.
Optimum buffer sizes for synchronous transfers are something less than the
SO_SNDBUF and SO_RCVBUF values (returned by a call to GetSockOpt), these
have a default of 8192, but a value of 16386 or 32672 are sometimes used for
bulck transfers like used by network back-up programs. But as you see these
are far below what OP is trying to use.

Willy.

[Sharon]

Re: TcpClient buffer size limit?

Ok, I did some tests and here is the results:
The Socket ReceiveBuffer and SendBuffer can be set by:
int block size = 8192 or 65536 or 131072 or 1048576 etc.
rvcSocket.SetSo cketOption(Sock etOptionLevel.S ocket,
SocketOptionNam e.ReceiveBuffer , blockSize);
and
sndSocket.SetSo cketOption(Sock etOptionLevel.S ocket,
SocketOptionNam e.SendBuffer, blockSize);

byte buffer = new byte[8192 or 65536 or 131072 or 1048576 etc.];
The Socket.Send(buf fer) does split the buffer and does multiple send to
ensure the complete buffer transmission.
Nevertheless, if the buffer is tool large (like 100Mbyte in my case), the
sending and receiving will throw an exception.

I tried to find the size limit that cause the exception, but with no success.
And this is a very bad thing for me. Why?!
- Because if at run time I'll catch the exception and do buffer splitting
over and over till no exception is thrown, I'll will loose to much time that
I do not have. So in order to save time I need to do it only once, and use
this found buffer size for all image frame I'm sending. The trouble with this
solution is that a single peak (down peak) will result in a small blocks
transmission for all future frames, and that is no good ether. So maybe I'll
combine the two methods by calculating the buffer size by catching the too
large buffer exception every time interval that also need to be figured out.

So that's what I found.

Any other idea will be welcome.


-------
Thanks
Sharon

[Willy Denoyette [MVP]]

Re: TcpClient buffer size limit?


"Sharon" <Sharon@discuss ions.microsoft. com> wrote in message
news:5C05307F-A1AC-426B-A60A-2F5408C1DF1D@mi crosoft.com...[color=blue]
> Ok, I did some tests and here is the results:
> The Socket ReceiveBuffer and SendBuffer can be set by:
> int block size = 8192 or 65536 or 131072 or 1048576 etc.
> rvcSocket.SetSo cketOption(Sock etOptionLevel.S ocket,
> SocketOptionNam e.ReceiveBuffer , blockSize);
> and
> sndSocket.SetSo cketOption(Sock etOptionLevel.S ocket,
> SocketOptionNam e.SendBuffer, blockSize);
>
> byte buffer = new byte[8192 or 65536 or 131072 or 1048576 etc.];
> The Socket.Send(buf fer) does split the buffer and does multiple send to
> ensure the complete buffer transmission.
> Nevertheless, if the buffer is tool large (like 100Mbyte in my case), the
> sending and receiving will throw an exception.
>
> I tried to find the size limit that cause the exception, but with no
> success.
> And this is a very bad thing for me. Why?!
> - Because if at run time I'll catch the exception and do buffer splitting
> over and over till no exception is thrown, I'll will loose to much time
> that
> I do not have. So in order to save time I need to do it only once, and use
> this found buffer size for all image frame I'm sending. The trouble with
> this
> solution is that a single peak (down peak) will result in a small blocks
> transmission for all future frames, and that is no good ether. So maybe
> I'll
> combine the two methods by calculating the buffer size by catching the too
> large buffer exception every time interval that also need to be figured
> out.
>
> So that's what I found.
>
> Any other idea will be welcome.
>
>
> -------
> Thanks
> Sharon[/color]

Setting the buffer size like you are doing by SetSocketOption to a value
which is larger than 64Kb is only possible when running on XP or W2K3 and
only when connecting to other systems that support RFC 1323 Window Scaling,
and only when the other system accepts this value during initial connect.
Note that setting a value larger than 64Kb will silently be accepted without
an error indication, however, the effective size will be lower than
requested.
I'm still not clear what you like to achieve, you are telling us how you are
doing things, but not why you are doing them, what issues do you have, what
is the context you are running in, what kind of connection are you using
(speed, latency...). I'm also unclear about your code, it looks like you are
using simple synchronous socket IO instead of applying an asynchronous
application design, maybe that's the reason why you are so eager to extend
the application buffer.
As I told you before using the default per interface type TCP registry
settings and an application buffer size of 64K you can easily saturate a
single Gigabit network interface.

Willy.

Willy.

[Sharon]

Re: TcpClient buffer size limit?

Hi Willy,

Ok' I'll try to make my case clearer:
I'm developing an application that uses a Frame Grabber that produce 100
mega byte frames.
The application need to distribute these frames to several computers, frame1
to pc1, frame2 to pc2, frame3 to pc3 and so on.
The frames are generated in very high speed, therefore the frames
transmission must also be very fast.
So I'm looking for the optimum TCP connection and configuration to achieve
the fastest speed.
We are also using (in the near future) GLAN, dual Xeon PC's with Windows XP.

From the tests I did, I found that a bigger TCP buffer making the
transmission faster (of course there is a limit for that).

Yes, I'm using simple synchronous socket IO and not an asynchronous. Why
should I prefer the the asynchronous transmission? Will it make the frames
transmission faster?

I'm not sure I understand what you mean when you say "using the default per
interface type TCP registry settings and an application buffer size of 64K
you can easily saturate a single Gigabit network interface."
Can you please elaborate on that?

Any suggestion will be welcome.

-----------
Thanks
Sharon