Http stream performance

[Leon Jollans]

Hi.
I'm reading a 5 meg (or so) file from the response stream of a WebRequest,
and it's pitifully slow. I mean *really* slow, in the order of 15 minutes to
download. Now obviously I'm caching this on the client (Windows app /
IsolatedStorage ), but I can't have it taking this long to download in the
first instance. We've got Gigabit LAN. what's the bottleneck?

I could make the resource available via a different protocol / means if
necessary to overcome the problem.

Has anyone had similar issues or know of a better method of large network
downloads?

Cheers,
Leon

[Jon Skeet]

Re: Http stream performance

Leon Jollans <Leon.JollansRE MOVE__THIS@xama n.com> wrote:[color=blue]
> I'm reading a 5 meg (or so) file from the response stream of a WebRequest,
> and it's pitifully slow. I mean *really* slow, in the order of 15 minutes to
> download. Now obviously I'm caching this on the client (Windows app /
> IsolatedStorage ), but I can't have it taking this long to download in the
> first instance. We've got Gigabit LAN. what's the bottleneck?[/color]

How are you reading it, exactly? (Please show some code, in other
words.) Have you tried reading it with a plain browser (or wget) to see
whether it's the client or the server that's holding it up?

--
Jon Skeet - <skeet@pobox.co m>

http://www.pobox.com/~skeet

If replying to the group, please do not mail me too

[Leon Jollans]

Re: Http stream performance

It was definitely the client all along. but I think I've sussed it now: I
was reading the stream byte by byte into a pre-initialised array using
ReadByte from the stream directly and checking for -1 each time.

now I'm doing it using a BinaryReader in chunks and the speed is as per
expectations. code below for reference.

void readStream(Stre am stream, long contentLength){
int i=0;
byte[] bytes;
BinaryReader reader = new BinaryReader(st ream);
MemoryStream ms = new MemoryStream((i nt)contentLengt h);
do
{
bytes = reader.ReadByte s(128);
if(bytes.Length > 0)
{
ms.Write(bytes, 0,bytes.Length) ;
}
else{
break;
}
}while( true );
reader.Close();
stream.Close();
ms.Flush();
bytes = ms.ToArray();
ms.Close();

...

cheers

Leon



"Jon Skeet" <skeet@pobox.co m> wrote in message
news:MPG.19ca81 e8158aff7098969 3@news.microsof t.com...[color=blue]
> Leon Jollans <Leon.JollansRE MOVE__THIS@xama n.com> wrote:[color=green]
> > I'm reading a 5 meg (or so) file from the response stream of a[/color][/color]
WebRequest,[color=blue][color=green]
> > and it's pitifully slow. I mean *really* slow, in the order of 15[/color][/color]
minutes to[color=blue][color=green]
> > download. Now obviously I'm caching this on the client (Windows app /
> > IsolatedStorage ), but I can't have it taking this long to download in[/color][/color]
the[color=blue][color=green]
> > first instance. We've got Gigabit LAN. what's the bottleneck?[/color]
>
> How are you reading it, exactly? (Please show some code, in other
> words.) Have you tried reading it with a plain browser (or wget) to see
> whether it's the client or the server that's holding it up?
>
> --
> Jon Skeet - <skeet@pobox.co m>
> http://www.pobox.com/~skeet
> If replying to the group, please do not mail me too[/color]

[Jon Skeet]

Re: Http stream performance

Leon Jollans <Leon.JollansRE MOVE__THIS@xama n.com> wrote:[color=blue]
> It was definitely the client all along. but I think I've sussed it now: I
> was reading the stream byte by byte into a pre-initialised array using
> ReadByte from the stream directly and checking for -1 each time.[/color]

Yes, that would definitely slow it down :)
[color=blue]
> now I'm doing it using a BinaryReader in chunks and the speed is as per
> expectations. code below for reference.
>
> void readStream(Stre am stream, long contentLength){
> int i=0;
> byte[] bytes;
> BinaryReader reader = new BinaryReader(st ream);
> MemoryStream ms = new MemoryStream((i nt)contentLengt h);
> do
> {
> bytes = reader.ReadByte s(128);
> if(bytes.Length > 0)
> {
> ms.Write(bytes, 0,bytes.Length) ;
> }
> else{
> break;
> }
> }while( true );
> reader.Close();
> stream.Close();
> ms.Flush();
> bytes = ms.ToArray();
> ms.Close();[/color]

Rather than doing that, I'd do something like:
(assuming you can make contentLength an int instead of a long)


byte[] data = new byte[contentLength];

int offset=0;
int remaining = contentLength;

while (remaining > 0)
{
int read = stream.Read (data, offset, remaining);
if (read==-1)
// Up to you - throw an exception or something similar
// because we didn't get as much as we expected
offset+=read;
remaining-=read;
}

--
Jon Skeet - <skeet@pobox.co m>

http://www.pobox.com/~skeet

If replying to the group, please do not mail me too

[Leon Jollans]

Re: Http stream performance

That's much faster! thanks :)

Leon

"Jon Skeet" <skeet@pobox.co m> wrote in message
news:MPG.19ca98 e0f1ed727598969 6@news.microsof t.com...[color=blue]
> Leon Jollans <Leon.JollansRE MOVE__THIS@xama n.com> wrote:[color=green]
> > It was definitely the client all along. but I think I've sussed it now:[/color][/color]
I[color=blue][color=green]
> > was reading the stream byte by byte into a pre-initialised array using
> > ReadByte from the stream directly and checking for -1 each time.[/color]
>
> Yes, that would definitely slow it down :)
>[color=green]
> > now I'm doing it using a BinaryReader in chunks and the speed is as per
> > expectations. code below for reference.
> >
> > void readStream(Stre am stream, long contentLength){
> > int i=0;
> > byte[] bytes;
> > BinaryReader reader = new BinaryReader(st ream);
> > MemoryStream ms = new MemoryStream((i nt)contentLengt h);
> > do
> > {
> > bytes = reader.ReadByte s(128);
> > if(bytes.Length > 0)
> > {
> > ms.Write(bytes, 0,bytes.Length) ;
> > }
> > else{
> > break;
> > }
> > }while( true );
> > reader.Close();
> > stream.Close();
> > ms.Flush();
> > bytes = ms.ToArray();
> > ms.Close();[/color]
>
> Rather than doing that, I'd do something like:
> (assuming you can make contentLength an int instead of a long)
>
>
> byte[] data = new byte[contentLength];
>
> int offset=0;
> int remaining = contentLength;
>
> while (remaining > 0)
> {
> int read = stream.Read (data, offset, remaining);
> if (read==-1)
> // Up to you - throw an exception or something similar
> // because we didn't get as much as we expected
> offset+=read;
> remaining-=read;
> }
>
> --
> Jon Skeet - <skeet@pobox.co m>
> http://www.pobox.com/~skeet
> If replying to the group, please do not mail me too[/color]