Troubleshooting ASP.Net memory usage issues?

[Daniel Peterson]

I'm responsible for a pair of IIS 6 webservers that run our production
ASP.Net application. We push code on a monthly basis, and about two weeks
after our October code push, we started to run into occasional webserver
errors.

The application pool is configured to run as a domain user which has
permissions to the web code. When we have issues, the webserver will return
the ASP.Net "Server Application Unavailable" page, and our app pool memory
usage is really high (anywhere from 2 to 4 times our normal usage). Non
asp.net content is served up just fine. If I recycle the application pool
or do an iisreset, memory usage drops back to normal and the site starts
working again.

Nothing is logged to the eventlog or http error log, and the only errors we
see from our application has been the occasional "Exception of type
System.OutOfMem oryException was thrown." The same page doesn't error every
time, and the function I've seen throw exceptions is incredibly generic and
used heavily throughout our entire application (or so our devs have told
me). Our application doesn't even log an error every time this has
happened.

How should I go about troubleshooting this? Our devs are saying it's a
server issue, but I'm leaning towards a code issue. These servers are
running Windows 2003 Server standard, fully updated as of this last weekend.

[Eliyahu Goldin]

Re: Troubleshooting ASP.Net memory usage issues?

What session management are you using. InProc sessions are major memory
consumers.

--
Eliyahu Goldin,
Software Developer
Microsoft MVP [ASP.NET]

http://msmvps.com/blogs/egoldin

http://usableasp.net

"Daniel Peterson" <pythas@hotmail .comwrote in message
news:CB009C9F-B38F-4937-8AF9-256502E90CDE@mi crosoft.com...

I'm responsible for a pair of IIS 6 webservers that run our production
ASP.Net application. We push code on a monthly basis, and about two weeks
after our October code push, we started to run into occasional webserver
errors.
>
The application pool is configured to run as a domain user which has
permissions to the web code. When we have issues, the webserver will
return the ASP.Net "Server Application Unavailable" page, and our app pool
memory usage is really high (anywhere from 2 to 4 times our normal usage).
Non asp.net content is served up just fine. If I recycle the application
pool or do an iisreset, memory usage drops back to normal and the site
starts working again.
>
Nothing is logged to the eventlog or http error log, and the only errors
we see from our application has been the occasional "Exception of type
System.OutOfMem oryException was thrown." The same page doesn't error
every time, and the function I've seen throw exceptions is incredibly
generic and used heavily throughout our entire application (or so our devs
have told me). Our application doesn't even log an error every time this
has happened.
>
How should I go about troubleshooting this? Our devs are saying it's a
server issue, but I'm leaning towards a code issue. These servers are
running Windows 2003 Server standard, fully updated as of this last
weekend.

[Kevin Spencer]

Re: Troubleshooting ASP.Net memory usage issues?

Hi Daniel,

It's a code issue. It will be one of 2 possible reasons: 1. Memory Leak.
Yes, .Net applications can leak memory, if they use unmanaged resources that
are not disposed properly. 2. Memory Fragmentation. If the same block of
memory is re-used as, for example, an application-level collection that has
members added and removed over a long period of time, that memory can become
fragmented, necessitating the allocation of more memory to create contiguous
blocks when they are needed.

--
HTH,

Kevin Spencer
Chicken Salad Surgeon
Microsoft MVP

"Daniel Peterson" <pythas@hotmail .comwrote in message
news:CB009C9F-B38F-4937-8AF9-256502E90CDE@mi crosoft.com...

I'm responsible for a pair of IIS 6 webservers that run our production
ASP.Net application. We push code on a monthly basis, and about two weeks
after our October code push, we started to run into occasional webserver
errors.
>
The application pool is configured to run as a domain user which has
permissions to the web code. When we have issues, the webserver will
return the ASP.Net "Server Application Unavailable" page, and our app pool
memory usage is really high (anywhere from 2 to 4 times our normal usage).
Non asp.net content is served up just fine. If I recycle the application
pool or do an iisreset, memory usage drops back to normal and the site
starts working again.
>
Nothing is logged to the eventlog or http error log, and the only errors
we see from our application has been the occasional "Exception of type
System.OutOfMem oryException was thrown." The same page doesn't error
every time, and the function I've seen throw exceptions is incredibly
generic and used heavily throughout our entire application (or so our devs
have told me). Our application doesn't even log an error every time this
has happened.
>
How should I go about troubleshooting this? Our devs are saying it's a
server issue, but I'm leaning towards a code issue. These servers are
running Windows 2003 Server standard, fully updated as of this last
weekend.

[Daniel Peterson]

Re: Troubleshooting ASP.Net memory usage issues?

We're using the session state db.

"Eliyahu Goldin" <REMOVEALLCAPIT ALSeEgGoldDinN@ mMvVpPsS.orgwro te in
message news:e0N$Dc2JIH A.2064@TK2MSFTN GP06.phx.gbl...

What session management are you using. InProc sessions are major memory
consumers.
>
--
Eliyahu Goldin,
Software Developer
Microsoft MVP [ASP.NET]

http://msmvps.com/blogs/egoldin

http://usableasp.net

>
>
"Daniel Peterson" <pythas@hotmail .comwrote in message
news:CB009C9F-B38F-4937-8AF9-256502E90CDE@mi crosoft.com...

>I'm responsible for a pair of IIS 6 webservers that run our production
>ASP.Net application. We push code on a monthly basis, and about two
>weeks after our October code push, we started to run into occasional
>webserver errors.
>>
>The application pool is configured to run as a domain user which has
>permissions to the web code. When we have issues, the webserver will
>return the ASP.Net "Server Application Unavailable" page, and our app
>pool memory usage is really high (anywhere from 2 to 4 times our normal
>usage). Non asp.net content is served up just fine. If I recycle the
>application pool or do an iisreset, memory usage drops back to normal and
>the site starts working again.
>>
>Nothing is logged to the eventlog or http error log, and the only errors
>we see from our application has been the occasional "Exception of type
>System.OutOfMe moryException was thrown." The same page doesn't error
>every time, and the function I've seen throw exceptions is incredibly
>generic and used heavily throughout our entire application (or so our
>devs have told me). Our application doesn't even log an error every time
>this has happened.
>>
>How should I go about troubleshooting this? Our devs are saying it's a
>server issue, but I'm leaning towards a code issue. These servers are
>running Windows 2003 Server standard, fully updated as of this last
>weekend.

>
>

[Daniel Peterson]

Re: Troubleshooting ASP.Net memory usage issues?

So what's the best route to troubleshoot, just bust out debugdiag and go
from there?

If it's option #2 as you mention below, would cycling the application pool
more frequently solve that? I think we're using the 1740 minute default
right now.

"Kevin Spencer" <unclechutney@n othinks.comwrot e in message
news:#YiaFK4JIH A.5360@TK2MSFTN GP03.phx.gbl...

Hi Daniel,
>
It's a code issue. It will be one of 2 possible reasons: 1. Memory Leak.
Yes, .Net applications can leak memory, if they use unmanaged resources
that are not disposed properly. 2. Memory Fragmentation. If the same block
of memory is re-used as, for example, an application-level collection that
has members added and removed over a long period of time, that memory can
become fragmented, necessitating the allocation of more memory to create
contiguous blocks when they are needed.
>
--
HTH,
>
Kevin Spencer
Chicken Salad Surgeon
Microsoft MVP
>
"Daniel Peterson" <pythas@hotmail .comwrote in message
news:CB009C9F-B38F-4937-8AF9-256502E90CDE@mi crosoft.com...

>I'm responsible for a pair of IIS 6 webservers that run our production
>ASP.Net application. We push code on a monthly basis, and about two
>weeks after our October code push, we started to run into occasional
>webserver errors.
>>
>The application pool is configured to run as a domain user which has
>permissions to the web code. When we have issues, the webserver will
>return the ASP.Net "Server Application Unavailable" page, and our app
>pool memory usage is really high (anywhere from 2 to 4 times our normal
>usage). Non asp.net content is served up just fine. If I recycle the
>application pool or do an iisreset, memory usage drops back to normal and
>the site starts working again.
>>
>Nothing is logged to the eventlog or http error log, and the only errors
>we see from our application has been the occasional "Exception of type
>System.OutOfMe moryException was thrown." The same page doesn't error
>every time, and the function I've seen throw exceptions is incredibly
>generic and used heavily throughout our entire application (or so our
>devs have told me). Our application doesn't even log an error every time
>this has happened.
>>
>How should I go about troubleshooting this? Our devs are saying it's a
>server issue, but I'm leaning towards a code issue. These servers are
>running Windows 2003 Server standard, fully updated as of this last
>weekend.

>
>

[Kevin Spencer]

Re: Troubleshooting ASP.Net memory usage issues?

Hi Daniel,

You could set up some performance counters, but the first thing I think I
would do is look for evidence of the 2 conditions I specified in the code.
It might very well show up rather easily if you know what you're looking
for.

--
HTH,

Kevin Spencer
Chicken Salad Surgeon
Microsoft MVP

"Daniel Peterson" <pythas@hotmail .comwrote in message
news:0323BA74-E5C3-4FEA-9C05-38B520DA602D@mi crosoft.com...

So what's the best route to troubleshoot, just bust out debugdiag and go
from there?
>
If it's option #2 as you mention below, would cycling the application pool
more frequently solve that? I think we're using the 1740 minute default
right now.
>
"Kevin Spencer" <unclechutney@n othinks.comwrot e in message
news:#YiaFK4JIH A.5360@TK2MSFTN GP03.phx.gbl...

>Hi Daniel,
>>
>It's a code issue. It will be one of 2 possible reasons: 1. Memory Leak.
>Yes, .Net applications can leak memory, if they use unmanaged resources
>that are not disposed properly. 2. Memory Fragmentation. If the same
>block of memory is re-used as, for example, an application-level
>collection that has members added and removed over a long period of time,
>that memory can become fragmented, necessitating the allocation of more
>memory to create contiguous blocks when they are needed.
>>
>--
>HTH,
>>
>Kevin Spencer
>Chicken Salad Surgeon
>Microsoft MVP
>>
>"Daniel Peterson" <pythas@hotmail .comwrote in message
>news:CB009C9 F-B38F-4937-8AF9-256502E90CDE@mi crosoft.com...

>>I'm responsible for a pair of IIS 6 webservers that run our production
>>ASP.Net application. We push code on a monthly basis, and about two
>>weeks after our October code push, we started to run into occasional
>>webserver errors.
>>>
>>The application pool is configured to run as a domain user which has
>>permissions to the web code. When we have issues, the webserver will
>>return the ASP.Net "Server Application Unavailable" page, and our app
>>pool memory usage is really high (anywhere from 2 to 4 times our normal
>>usage). Non asp.net content is served up just fine. If I recycle the
>>application pool or do an iisreset, memory usage drops back to normal
>>and the site starts working again.
>>>
>>Nothing is logged to the eventlog or http error log, and the only errors
>>we see from our application has been the occasional "Exception of type
>>System.OutOfM emoryException was thrown." The same page doesn't error
>>every time, and the function I've seen throw exceptions is incredibly
>>generic and used heavily throughout our entire application (or so our
>>devs have told me). Our application doesn't even log an error every
>>time this has happened.
>>>
>>How should I go about troubleshooting this? Our devs are saying it's a
>>server issue, but I'm leaning towards a code issue. These servers are
>>running Windows 2003 Server standard, fully updated as of this last
>>weekend.

>>
>>

[Scott Roberts]

Re: Troubleshooting ASP.Net memory usage issues?

I agree with Kevin, it's a code issue. Too many coders think they don't have
to worry about memory management anymore since "DotNet does it for me!".

Personally, I would start by looking at the Session and/or Application
object. It's likely that some coder decided it would be convenient to stuff
a huge collection of search results (or some such thing) into the session
object and now it's being held there until the session expires.

You could set up some performance counters, but the first thing I think I
would do is look for evidence of the 2 conditions I specified in the code.
It might very well show up rather easily if you know what you're looking
for.

[Dave Bush]

Re: Troubleshooting ASP.Net memory usage issues?

Another culprit could be IDisposable objects not being disposed. You
doing anything with the graphics object maybe?

Dave Bush

http://blog.dmbcllc.com

-----Original Message-----
From: Scott Roberts [mailto:sroberts @no.spam.here-webworks-software.com]

Posted At: Friday, November 16, 2007 9:34 AM
Posted To: microsoft.publi c.dotnet.framew ork.aspnet
Conversation: Troubleshooting ASP.Net memory usage issues?
Subject: Re: Troubleshooting ASP.Net memory usage issues?

I agree with Kevin, it's a code issue. Too many coders think they don't
have
to worry about memory management anymore since "DotNet does it for me!".

Personally, I would start by looking at the Session and/or Application
object. It's likely that some coder decided it would be convenient to
stuff
a huge collection of search results (or some such thing) into the
session
object and now it's being held there until the session expires.

You could set up some performance counters, but the first thing I

think I

would do is look for evidence of the 2 conditions I specified in the

code.

It might very well show up rather easily if you know what you're

looking

for.

[Daniel Peterson]

Re: Troubleshooting ASP.Net memory usage issues?

Thanks for your help everyone. We found a 3rd party component with
debugdiag that appears to be leaking memory, although our devs seemed to
dismiss that as the issue, and they're still pointing to the servers. If
they don't want to look, I'm not sure how much more I can do.

They think that

http://support.microsoft.com/default.aspx?scid=kb;en-us;821268&Product=asp

this is what's going on, although we're not seeing any of those errors or
eventlog messages.

Ugh.

"Scott Roberts" <sroberts@no.sp am.here-webworks-software.comwro te in
message news:etWOA3FKIH A.5208@TK2MSFTN GP04.phx.gbl...

I agree with Kevin, it's a code issue. Too many coders think they don't
have to worry about memory management anymore since "DotNet does it for
me!".
>
Personally, I would start by looking at the Session and/or Application
object. It's likely that some coder decided it would be convenient to
stuff a huge collection of search results (or some such thing) into the
session object and now it's being held there until the session expires.
>

>You could set up some performance counters, but the first thing I think I
>would do is look for evidence of the 2 conditions I specified in the
>code. It might very well show up rather easily if you know what you're
>looking for.

>

[Scott Roberts]

Re: Troubleshooting ASP.Net memory usage issues?

Thanks for your help everyone. We found a 3rd party component with

debugdiag that appears to be leaking memory, although our devs seemed to
dismiss that as the issue, and they're still pointing to the servers. If
they don't want to look, I'm not sure how much more I can do.

In your original post you mentioned that the problem started after a new
version was published, right? Was this 3rd party component used in the old
version of the site? Was it upgraded for the new version? Does the new
version use some features of this component that were previously unused?

They think that

http://support.microsoft.com/default.aspx?scid=kb;en-us;821268&Product=asp

this is what's going on, although we're not seeing any of those errors or
eventlog messages.

I don't see anything pertaining to large memory usage in that KB. Is it a
memory problem or a thread pool problem?

[Daniel Peterson]

Re: Troubleshooting ASP.Net memory usage issues?

To answer your questions:

1. Yes, we noticed the issues about 2 weeks after a release. The 3rd party
component was updated in that release, I don't think we're using any new
features, but I'll check.

2. I don't see anything in that KB article that describes the symptoms
we're seeing either. We're not getting any of the error messages that would
suggest a thread pool issue, I'm not quite sure how they ended up at that kb
article.

"Scott Roberts" <sroberts@no.sp am.here-webworks-software.comwro te in
message news:O1RhfZ5KIH A.4688@TK2MSFTN GP06.phx.gbl...

>Thanks for your help everyone. We found a 3rd party component with
>debugdiag that appears to be leaking memory, although our devs seemed to
>dismiss that as the issue, and they're still pointing to the servers. If
>they don't want to look, I'm not sure how much more I can do.

>
In your original post you mentioned that the problem started after a new
version was published, right? Was this 3rd party component used in the old
version of the site? Was it upgraded for the new version? Does the new
version use some features of this component that were previously unused?
>

>They think that
>http://support.microsoft.com/default...68&Product=asp
>this is what's going on, although we're not seeing any of those errors or
>eventlog messages.

>
I don't see anything pertaining to large memory usage in that KB. Is it a
memory problem or a thread pool problem?