GetRequestStream slows down?

[Jim Toth]

I have a multithreaded application that is posting XML requests to a remote
web site. As more and more calls are made to
HttpWebRequest. GetRequestStrea m, the length of time increases for each call
from an initial 2-3 seconds to anywhere from 15-30 seconds. What am I doing
that is causing this method call's duration to increase and how do I make it
run faster? Below is some pseudo-code for the method that is called in a
multi-threaded fashion...

- Jim

private void MultithreadedMe thod()
{
string url = ConfigurationMa nager.AppSettin gs["providerUr l"];

request = (HttpWebRequest )WebRequest.Cre ate(url);
request.Method = "POST";
request.Content Type = "text/xml";

startCount = Environment.Tic kCount;
using (Stream s = request.GetRequ estStream())
{
elapsedCount = Environment.Tic kCount - startCount;
//the elapsed time increases with each call
EventLog.WriteE ntry("request.G etRequestStream () = " + elapsedCount,
EventLogEntryTy pe.Information) ;

using (StreamWriter writer = new StreamWriter(s) )
{
writer.WriteLin e(requestXml);
writer.Flush();
writer.Close();
}

s.Close();
}


request.Timeout = requestTimeoutM illiseconds;
response = request.GetResp onse();

//handle the response here
}

[John Murray]

Re: GetRequestStrea m slows down?

Are these calls concurrent or consecutive. In 1.1 these requests were
made on the thread pool, so you might be running into some concurrency
issues.

Additionally, depending on the size of the page returned, you might also
be running in to GC issues -- so for example, if the content returned is
over 85k (assuming it being put into one big string variable at some
point) that memory has the potential of being put into the large object
heap which is only swept and not compacted .... so you might be running
into some memory issues.

Also, the s.Close is redundant and repetitive ... it's getting called in
IDispose after you exit from your using block.

Also, make sure that you are closing the response object that you get
back from request.GetResp onse(); .... not doing so can lead to some
unmitigated nastiness

John



Jim Toth wrote:[color=blue]
> I have a multithreaded application that is posting XML requests to a remote
> web site. As more and more calls are made to
> HttpWebRequest. GetRequestStrea m, the length of time increases for each call
> from an initial 2-3 seconds to anywhere from 15-30 seconds. What am I doing
> that is causing this method call's duration to increase and how do I make it
> run faster? Below is some pseudo-code for the method that is called in a
> multi-threaded fashion...
>
> - Jim
>
> private void MultithreadedMe thod()
> {
> string url = ConfigurationMa nager.AppSettin gs["providerUr l"];
>
> request = (HttpWebRequest )WebRequest.Cre ate(url);
> request.Method = "POST";
> request.Content Type = "text/xml";
>
> startCount = Environment.Tic kCount;
> using (Stream s = request.GetRequ estStream())
> {
> elapsedCount = Environment.Tic kCount - startCount;
> //the elapsed time increases with each call
> EventLog.WriteE ntry("request.G etRequestStream () = " + elapsedCount,
> EventLogEntryTy pe.Information) ;
>
> using (StreamWriter writer = new StreamWriter(s) )
> {
> writer.WriteLin e(requestXml);
> writer.Flush();
> writer.Close();
> }
>
> s.Close();
> }
>
>
> request.Timeout = requestTimeoutM illiseconds;
> response = request.GetResp onse();
>
> //handle the response here
> }
>[/color]

[Peter Bromberg [C# MVP]]

RE: GetRequestStrea m slows down?

Aside from the other comments, the title of your method "MultithreadedM ethod"
is confusing, as it isn't multithreaded at all the way it is written.
If you really have a lot of this going on, you probably want to look at the
asynchronous BeginXXX / EndXXX overloads which use the Threadpool "under the
hood" - or use a custom threadpool.
Peter

--
Co-founder, Eggheadcafe.com developer portal:

http://www.eggheadcafe.com

UnBlog:

http://petesbloggerama.blogspot.com

"Jim Toth" wrote:
[color=blue]
> I have a multithreaded application that is posting XML requests to a remote
> web site. As more and more calls are made to
> HttpWebRequest. GetRequestStrea m, the length of time increases for each call
> from an initial 2-3 seconds to anywhere from 15-30 seconds. What am I doing
> that is causing this method call's duration to increase and how do I make it
> run faster? Below is some pseudo-code for the method that is called in a
> multi-threaded fashion...
>
> - Jim
>
> private void MultithreadedMe thod()
> {
> string url = ConfigurationMa nager.AppSettin gs["providerUr l"];
>
> request = (HttpWebRequest )WebRequest.Cre ate(url);
> request.Method = "POST";
> request.Content Type = "text/xml";
>
> startCount = Environment.Tic kCount;
> using (Stream s = request.GetRequ estStream())
> {
> elapsedCount = Environment.Tic kCount - startCount;
> //the elapsed time increases with each call
> EventLog.WriteE ntry("request.G etRequestStream () = " + elapsedCount,
> EventLogEntryTy pe.Information) ;
>
> using (StreamWriter writer = new StreamWriter(s) )
> {
> writer.WriteLin e(requestXml);
> writer.Flush();
> writer.Close();
> }
>
> s.Close();
> }
>
>
> request.Timeout = requestTimeoutM illiseconds;
> response = request.GetResp onse();
>
> //handle the response here
> }
>[/color]

[Jim Toth]

RE: GetRequestStrea m slows down?

What I meant by "MultithreadedM ethod" is that the method is called many times
on many different threads.

I've seen the asynchronous calls for GetRequestStrea m... will they get me
around potential contention issues mentioned?

"Peter Bromberg [C# MVP]" wrote:
[color=blue]
> Aside from the other comments, the title of your method "MultithreadedM ethod"
> is confusing, as it isn't multithreaded at all the way it is written.
> If you really have a lot of this going on, you probably want to look at the
> asynchronous BeginXXX / EndXXX overloads which use the Threadpool "under the
> hood" - or use a custom threadpool.
> Peter
>
> --
> Co-founder, Eggheadcafe.com developer portal:
> http://www.eggheadcafe.com
> UnBlog:
> http://petesbloggerama.blogspot.com
>
>
>
>
> "Jim Toth" wrote:
>[color=green]
> > I have a multithreaded application that is posting XML requests to a remote
> > web site. As more and more calls are made to
> > HttpWebRequest. GetRequestStrea m, the length of time increases for each call
> > from an initial 2-3 seconds to anywhere from 15-30 seconds. What am I doing
> > that is causing this method call's duration to increase and how do I make it
> > run faster? Below is some pseudo-code for the method that is called in a
> > multi-threaded fashion...
> >
> > - Jim
> >
> > private void MultithreadedMe thod()
> > {
> > string url = ConfigurationMa nager.AppSettin gs["providerUr l"];
> >
> > request = (HttpWebRequest )WebRequest.Cre ate(url);
> > request.Method = "POST";
> > request.Content Type = "text/xml";
> >
> > startCount = Environment.Tic kCount;
> > using (Stream s = request.GetRequ estStream())
> > {
> > elapsedCount = Environment.Tic kCount - startCount;
> > //the elapsed time increases with each call
> > EventLog.WriteE ntry("request.G etRequestStream () = " + elapsedCount,
> > EventLogEntryTy pe.Information) ;
> >
> > using (StreamWriter writer = new StreamWriter(s) )
> > {
> > writer.WriteLin e(requestXml);
> > writer.Flush();
> > writer.Close();
> > }
> >
> > s.Close();
> > }
> >
> >
> > request.Timeout = requestTimeoutM illiseconds;
> > response = request.GetResp onse();
> >
> > //handle the response here
> > }
> >[/color][/color]

[Peter Bromberg [C# MVP]]

RE: GetRequestStrea m slows down?

Correct, which is a misnomer. If it is called many different times, you
should probably be using the Threadpool and the asynchronous
IAsyncResult result = request.BeginGe tResponse(
new AsyncCallback(U pdateItem),stat e);

semantics. This will speed up your operation by an order of magnitude since
it offers parallelism (multiple requests in operation at the same time).
Peter

--
Co-founder, Eggheadcafe.com developer portal:

http://www.eggheadcafe.com

UnBlog:

http://petesbloggerama.blogspot.com

"Jim Toth" wrote:
[color=blue]
> What I meant by "MultithreadedM ethod" is that the method is called many times
> on many different threads.
>
> I've seen the asynchronous calls for GetRequestStrea m... will they get me
> around potential contention issues mentioned?
>
> "Peter Bromberg [C# MVP]" wrote:
>[color=green]
> > Aside from the other comments, the title of your method "MultithreadedM ethod"
> > is confusing, as it isn't multithreaded at all the way it is written.
> > If you really have a lot of this going on, you probably want to look at the
> > asynchronous BeginXXX / EndXXX overloads which use the Threadpool "under the
> > hood" - or use a custom threadpool.
> > Peter
> >
> > --
> > Co-founder, Eggheadcafe.com developer portal:
> > http://www.eggheadcafe.com
> > UnBlog:
> > http://petesbloggerama.blogspot.com
> >
> >
> >
> >
> > "Jim Toth" wrote:
> >[color=darkred]
> > > I have a multithreaded application that is posting XML requests to a remote
> > > web site. As more and more calls are made to
> > > HttpWebRequest. GetRequestStrea m, the length of time increases for each call
> > > from an initial 2-3 seconds to anywhere from 15-30 seconds. What am I doing
> > > that is causing this method call's duration to increase and how do I make it
> > > run faster? Below is some pseudo-code for the method that is called in a
> > > multi-threaded fashion...
> > >
> > > - Jim
> > >
> > > private void MultithreadedMe thod()
> > > {
> > > string url = ConfigurationMa nager.AppSettin gs["providerUr l"];
> > >
> > > request = (HttpWebRequest )WebRequest.Cre ate(url);
> > > request.Method = "POST";
> > > request.Content Type = "text/xml";
> > >
> > > startCount = Environment.Tic kCount;
> > > using (Stream s = request.GetRequ estStream())
> > > {
> > > elapsedCount = Environment.Tic kCount - startCount;
> > > //the elapsed time increases with each call
> > > EventLog.WriteE ntry("request.G etRequestStream () = " + elapsedCount,
> > > EventLogEntryTy pe.Information) ;
> > >
> > > using (StreamWriter writer = new StreamWriter(s) )
> > > {
> > > writer.WriteLin e(requestXml);
> > > writer.Flush();
> > > writer.Close();
> > > }
> > >
> > > s.Close();
> > > }
> > >
> > >
> > > request.Timeout = requestTimeoutM illiseconds;
> > > response = request.GetResp onse();
> > >
> > > //handle the response here
> > > }
> > >[/color][/color][/color]

[John Murray]

Re: GetRequestStrea m slows down?

If you are using the 1.1 or earlier version of the Framework, never make
calls on WebClient or WebRequest from within the threadpool -- as the
calls on those objects are also implemented using the threadpool (even
if you are calling them synchronously) ...... the result is a very high
potential for a deadlock.


Peter Bromberg [C# MVP] wrote:[color=blue]
> Correct, which is a misnomer. If it is called many different times, you
> should probably be using the Threadpool and the asynchronous
> IAsyncResult result = request.BeginGe tResponse(
> new AsyncCallback(U pdateItem),stat e);
>
> semantics. This will speed up your operation by an order of magnitude since
> it offers parallelism (multiple requests in operation at the same time).
> Peter
>[/color]

[Peter Bromberg [C# MVP]]

Re: GetRequestStrea m slows down?

Perhaps I should have stated my suggestion more clearly. What I meant was,
use the asynchronous BeginGetRespons e etc. method, which use the .NET
threadpool.
Peter

--
Co-founder, Eggheadcafe.com developer portal:

http://www.eggheadcafe.com

UnBlog:

http://petesbloggerama.blogspot.com

"John Murray" wrote:
[color=blue]
> If you are using the 1.1 or earlier version of the Framework, never make
> calls on WebClient or WebRequest from within the threadpool -- as the
> calls on those objects are also implemented using the threadpool (even
> if you are calling them synchronously) ...... the result is a very high
> potential for a deadlock.
>
>
> Peter Bromberg [C# MVP] wrote:[color=green]
> > Correct, which is a misnomer. If it is called many different times, you
> > should probably be using the Threadpool and the asynchronous
> > IAsyncResult result = request.BeginGe tResponse(
> > new AsyncCallback(U pdateItem),stat e);
> >
> > semantics. This will speed up your operation by an order of magnitude since
> > it offers parallelism (multiple requests in operation at the same time).
> > Peter
> >[/color]
>[/color]

[Jim Toth]

RE: GetRequestStrea m slows down?

It looks like I'm running into some issues with
ServicePointMan ager.DefaultCon nectionLimit, which is defaulted to 2, because
I'm sending data to the same web site URL over and over. I tried bumping
this up to 10, 15, and 20 and had to bump up the number of threads in the
ThreadPool as a result, however I'm seeing only marginal gains in
performance. Is there a configuration setting, either for .NET or Windows
Server 2003, that I should be looking at, or am I missing something else?

"Peter Bromberg [C# MVP]" wrote:
[color=blue]
> Correct, which is a misnomer. If it is called many different times, you
> should probably be using the Threadpool and the asynchronous
> IAsyncResult result = request.BeginGe tResponse(
> new AsyncCallback(U pdateItem),stat e);
>
> semantics. This will speed up your operation by an order of magnitude since
> it offers parallelism (multiple requests in operation at the same time).
> Peter
>
> --
> Co-founder, Eggheadcafe.com developer portal:
> http://www.eggheadcafe.com
> UnBlog:
> http://petesbloggerama.blogspot.com
>
>
>
>
> "Jim Toth" wrote:
>[color=green]
> > What I meant by "MultithreadedM ethod" is that the method is called many times
> > on many different threads.
> >
> > I've seen the asynchronous calls for GetRequestStrea m... will they get me
> > around potential contention issues mentioned?
> >
> > "Peter Bromberg [C# MVP]" wrote:
> >[color=darkred]
> > > Aside from the other comments, the title of your method "MultithreadedM ethod"
> > > is confusing, as it isn't multithreaded at all the way it is written.
> > > If you really have a lot of this going on, you probably want to look at the
> > > asynchronous BeginXXX / EndXXX overloads which use the Threadpool "under the
> > > hood" - or use a custom threadpool.
> > > Peter
> > >
> > > --
> > > Co-founder, Eggheadcafe.com developer portal:
> > > http://www.eggheadcafe.com
> > > UnBlog:
> > > http://petesbloggerama.blogspot.com
> > >
> > >
> > >
> > >
> > > "Jim Toth" wrote:
> > >
> > > > I have a multithreaded application that is posting XML requests to a remote
> > > > web site. As more and more calls are made to
> > > > HttpWebRequest. GetRequestStrea m, the length of time increases for each call
> > > > from an initial 2-3 seconds to anywhere from 15-30 seconds. What am I doing
> > > > that is causing this method call's duration to increase and how do I make it
> > > > run faster? Below is some pseudo-code for the method that is called in a
> > > > multi-threaded fashion...
> > > >
> > > > - Jim
> > > >
> > > > private void MultithreadedMe thod()
> > > > {
> > > > string url = ConfigurationMa nager.AppSettin gs["providerUr l"];
> > > >
> > > > request = (HttpWebRequest )WebRequest.Cre ate(url);
> > > > request.Method = "POST";
> > > > request.Content Type = "text/xml";
> > > >
> > > > startCount = Environment.Tic kCount;
> > > > using (Stream s = request.GetRequ estStream())
> > > > {
> > > > elapsedCount = Environment.Tic kCount - startCount;
> > > > //the elapsed time increases with each call
> > > > EventLog.WriteE ntry("request.G etRequestStream () = " + elapsedCount,
> > > > EventLogEntryTy pe.Information) ;
> > > >
> > > > using (StreamWriter writer = new StreamWriter(s) )
> > > > {
> > > > writer.WriteLin e(requestXml);
> > > > writer.Flush();
> > > > writer.Close();
> > > > }
> > > >
> > > > s.Close();
> > > > }
> > > >
> > > >
> > > > request.Timeout = requestTimeoutM illiseconds;
> > > > response = request.GetResp onse();
> > > >
> > > > //handle the response here
> > > > }
> > > >[/color][/color][/color]

[Peter Bromberg [C# MVP]]

RE: GetRequestStrea m slows down?

make sure you close the Response object along with any streams /
streamreaders each time. Aside from that, I've seen a lot of old newsgroups
threads on upping the DefaultConnecti onLimit with mixed results. It could be
at the server too, you know.
Peter

--
Co-founder, Eggheadcafe.com developer portal:

http://www.eggheadcafe.com

UnBlog:

http://petesbloggerama.blogspot.com

"Jim Toth" wrote:
[color=blue]
> I have a multithreaded application that is posting XML requests to a remote
> web site. As more and more calls are made to
> HttpWebRequest. GetRequestStrea m, the length of time increases for each call
> from an initial 2-3 seconds to anywhere from 15-30 seconds. What am I doing
> that is causing this method call's duration to increase and how do I make it
> run faster? Below is some pseudo-code for the method that is called in a
> multi-threaded fashion...
>
> - Jim
>
> private void MultithreadedMe thod()
> {
> string url = ConfigurationMa nager.AppSettin gs["providerUr l"];
>
> request = (HttpWebRequest )WebRequest.Cre ate(url);
> request.Method = "POST";
> request.Content Type = "text/xml";
>
> startCount = Environment.Tic kCount;
> using (Stream s = request.GetRequ estStream())
> {
> elapsedCount = Environment.Tic kCount - startCount;
> //the elapsed time increases with each call
> EventLog.WriteE ntry("request.G etRequestStream () = " + elapsedCount,
> EventLogEntryTy pe.Information) ;
>
> using (StreamWriter writer = new StreamWriter(s) )
> {
> writer.WriteLin e(requestXml);
> writer.Flush();
> writer.Close();
> }
>
> s.Close();
> }
>
>
> request.Timeout = requestTimeoutM illiseconds;
> response = request.GetResp onse();
>
> //handle the response here
> }
>[/color]