maximum .net process can use

[Zen]

Hi,

My production machine has 2G of memory, when aspnet_wp.exe goes up to about
~1.2G of memory usage, I start get out-of-memory exception. Other processes
don't use as much memory and I added all the peak memory usage of all the
processes (including aspnet_wp.exe), it goes up to no more than 1.5.

How is that possible? Would anyone know please help? thanks!

[Kevin Frey]

Re: maximum .net process can use

Firstly in Task Manager look at your "Commit Charge". This is the memory
committed to individual processes in total, plus the operating system's
memory requirements.

See how the "total" compares to the "limit". The limit will effectively be
the amount of physical RAM that you have plus your paging space.

"Zen" <zen@nononospam .com> wrote in message
news:%23IbVdJ9G GHA.2036@TK2MSF TNGP14.phx.gbl. ..[color=blue]
> Hi,
>
> My production machine has 2G of memory, when aspnet_wp.exe goes up to
> about ~1.2G of memory usage, I start get out-of-memory exception. Other
> processes don't use as much memory and I added all the peak memory usage
> of all the processes (including aspnet_wp.exe), it goes up to no more than
> 1.5.
>
> How is that possible? Would anyone know please help? thanks!
>[/color]

[Willy Denoyette [MVP]]

Re: maximum .net process can use


"Zen" <zen@nononospam .com> wrote in message
news:%23IbVdJ9G GHA.2036@TK2MSF TNGP14.phx.gbl. ..
| Hi,
|
| My production machine has 2G of memory, when aspnet_wp.exe goes up to
about
| ~1.2G of memory usage, I start get out-of-memory exception. Other
processes
| don't use as much memory and I added all the peak memory usage of all the
| processes (including aspnet_wp.exe), it goes up to no more than 1.5.
|
| How is that possible? Would anyone know please help? thanks!
|
|

The amount of RAM is one thing, but the most important is the amount of
Virtual memory your process has allocated.
Each windows process (32 bit not /4GT enabled) has a maximum of 2GB Virtual
Address Space (VAS). This space is used to map the code of the ASP.NET
worker the CLR the C runtime and a couple of other DLL's plus your
application code (the IL and JIT'd code) the native process heaps and the GC
heaps.
So when you say that aspnet_wp.exe goes up to ~1.2GB you have to know
exactly what you are looking at and what tool you are using for this.
Anyway, you should never assume you have 2GB available, you should profile
your application and have a close look at your allocation patterns and the
type of objects you are allocating, take care about large objects (>85kb)
these tend to fragment the large object heap (especially in v1.x), and you
should always be prepared to get OOM exceptions thrown on you especially in
server applications.

Willy.

[Zen]

Re: maximum .net process can use

I used taskmanager that came with window to look at the memory usage info.

Would it help if I add another G or 2 to the server? Apparently each process
can use only up to 2G max of memory, but other items including the OS itself
would occupy a big junk of total memory leaving less memory for the asp.net.
I don't know the answer the question to that because the following 2 items
contradicting to each other:
1) With virtual memory system and enough harddisk space, the asp.net process
is already virtually working with 2G of memory, so adding physical memory
would only speed up (less swapping with page faults) but woudn't help
reducing OOM exception likelihood. My max page file size on harddisk is
1.5G.
2) However, when I moved from 1G to 2G physical memory, the system no longer
ran into OOM as often as before. With 1G, it could never load all my data
into memory for processing without seeing OOMs. Apparently adding more
physical memory did help.

To rephrase my original question, if I can afford any memory size I want,
which amount would be the max I can benefit from? There must be a stopping
point somewhere. I thought it was 2G but I could be wrong and hoping that
I'm wrong because making my code to work with smaller memory amount would be
very costly.

thanks!
zeng

"Willy Denoyette [MVP]" <willy.denoyett e@telenet.be> wrote in message
news:eWT1WfCHGH A.3752@TK2MSFTN GP11.phx.gbl...[color=blue]
>
> "Zen" <zen@nononospam .com> wrote in message
> news:%23IbVdJ9G GHA.2036@TK2MSF TNGP14.phx.gbl. ..
> | Hi,
> |
> | My production machine has 2G of memory, when aspnet_wp.exe goes up to
> about
> | ~1.2G of memory usage, I start get out-of-memory exception. Other
> processes
> | don't use as much memory and I added all the peak memory usage of all
> the
> | processes (including aspnet_wp.exe), it goes up to no more than 1.5.
> |
> | How is that possible? Would anyone know please help? thanks!
> |
> |
>
> The amount of RAM is one thing, but the most important is the amount of
> Virtual memory your process has allocated.
> Each windows process (32 bit not /4GT enabled) has a maximum of 2GB
> Virtual
> Address Space (VAS). This space is used to map the code of the ASP.NET
> worker the CLR the C runtime and a couple of other DLL's plus your
> application code (the IL and JIT'd code) the native process heaps and the
> GC
> heaps.
> So when you say that aspnet_wp.exe goes up to ~1.2GB you have to know
> exactly what you are looking at and what tool you are using for this.
> Anyway, you should never assume you have 2GB available, you should profile
> your application and have a close look at your allocation patterns and the
> type of objects you are allocating, take care about large objects (>85kb)
> these tend to fragment the large object heap (especially in v1.x), and you
> should always be prepared to get OOM exceptions thrown on you especially
> in
> server applications.
>
> Willy.
>
>[/color]

[Zen]

Re: maximum .net process can use

There is no "Commit Charge" column to choose from in the Task Manager, I'm
using Windows2000.



"Kevin Frey" <kevin_g_frey@h otmail.com> wrote in message
news:OK6O259GGH A.1132@TK2MSFTN GP10.phx.gbl...[color=blue]
> Firstly in Task Manager look at your "Commit Charge". This is the memory
> committed to individual processes in total, plus the operating system's
> memory requirements.
>
> See how the "total" compares to the "limit". The limit will effectively be
> the amount of physical RAM that you have plus your paging space.
>
> "Zen" <zen@nononospam .com> wrote in message
> news:%23IbVdJ9G GHA.2036@TK2MSF TNGP14.phx.gbl. ..[color=green]
>> Hi,
>>
>> My production machine has 2G of memory, when aspnet_wp.exe goes up to
>> about ~1.2G of memory usage, I start get out-of-memory exception. Other
>> processes don't use as much memory and I added all the peak memory usage
>> of all the processes (including aspnet_wp.exe), it goes up to no more
>> than 1.5.
>>
>> How is that possible? Would anyone know please help? thanks!
>>[/color]
>
>[/color]

[Willy Denoyette [MVP]]

Re: maximum .net process can use


"Zen" <zen@nononospam .com> wrote in message
news:uRKqXqEHGH A.2012@TK2MSFTN GP14.phx.gbl...
|I used taskmanager that came with window to look at the memory usage info.
|
| Would it help if I add another G or 2 to the server?


No, it won't help you at all, unless you have a high paging rate.

Apparently each process
| can use only up to 2G max of memory, but other items including the OS
itself
| would occupy a big junk of total memory leaving less memory for the
asp.net.
| I don't know the answer the question to that because the following 2 items
| contradicting to each other:
| 1) With virtual memory system and enough harddisk space, the asp.net
process
| is already virtually working with 2G of memory, so adding physical memory
| would only speed up (less swapping with page faults) but woudn't help
| reducing OOM exception likelihood. My max page file size on harddisk is
| 1.5G.

Did you measure the paging rate? Your paging file is too small anyway, it
should be larger than physical memory, and it should be large enough to hold
all 'dirty' datapages of the total of all processes. That means that the OS
can never allocate more data segments (as the GC does) than the size of the
paging file, the system will OOM at allocation time when it's not able to
back-up the dirty pages to the paging file.




| 2) However, when I moved from 1G to 2G physical memory, the system no
longer
| ran into OOM as often as before. With 1G, it could never load all my data
| into memory for processing without seeing OOMs. Apparently adding more
| physical memory did help.
|

Yes, now I see why your page file is 1.5GB only, you didn't change it's size
wen you moved from 1GB to 2GB.

| To rephrase my original question, if I can afford any memory size I want,
| which amount would be the max I can benefit from? There must be a
stopping
| point somewhere. I thought it was 2G but I could be wrong and hoping that
| I'm wrong because making my code to work with smaller memory amount would
be
| very costly.
|

First thing to do is increase your page file size (make it 3GB), and watch
your paging consumption and the paging rate using perfmon (don't use taskman
for this). If you still incur OOM's, profile your application and check your
allocation patterns carefully, watch for containers like ArrayList, they
grow exponentially and the are a pain in the a** when they become larger
than 85KB. Watch your memory fragmentation level when profiling or
debugging. Memory fragmentation cannot be solved by adding memory.
Watch your unmanaged memory counters and your unmanaged resources, make sure
you are disposing correctly.
You should ONLY add RAM when the paging rate is realy too high and becomes a
performance bottleneck. Don't extend above 3GB if you can it's a waste of
money on 32 bit OS.


Willy.

[Zen]

Re: maximum .net process can use

Thanks for you input. Yes, I realized that my page file size was low too, I
was surprised that Win2000 didn't automatically change the page file size to
take advantage of the max page file size set. Occasionally my page swap
rate jumped up to 40-50/sec. I'm still wondering about the max size Windows
can utilize to serve a resource intensive process, a few things I found:
1) On Win2000, max page file is 4G so 4G physical max then.
2) On XP, there isn't limitation - I attempted to set it up to 12G and it
allowed me (I asked me to reboot and I haven't got a chance to do that
though). Does anyone know the limit? Assume that I have $ to max out XP
capability to avoid those page swaps.

-thinh


"Willy Denoyette [MVP]" <willy.denoyett e@telenet.be> wrote in message
news:uwMYnSFHGH A.528@TK2MSFTNG P12.phx.gbl...[color=blue]
>
> "Zen" <zen@nononospam .com> wrote in message
> news:uRKqXqEHGH A.2012@TK2MSFTN GP14.phx.gbl...
> |I used taskmanager that came with window to look at the memory usage
> info.
> |
> | Would it help if I add another G or 2 to the server?
>
>
> No, it won't help you at all, unless you have a high paging rate.
>
> Apparently each process
> | can use only up to 2G max of memory, but other items including the OS
> itself
> | would occupy a big junk of total memory leaving less memory for the
> asp.net.
> | I don't know the answer the question to that because the following 2
> items
> | contradicting to each other:
> | 1) With virtual memory system and enough harddisk space, the asp.net
> process
> | is already virtually working with 2G of memory, so adding physical
> memory
> | would only speed up (less swapping with page faults) but woudn't help
> | reducing OOM exception likelihood. My max page file size on harddisk is
> | 1.5G.
>
> Did you measure the paging rate? Your paging file is too small anyway, it
> should be larger than physical memory, and it should be large enough to
> hold
> all 'dirty' datapages of the total of all processes. That means that the
> OS
> can never allocate more data segments (as the GC does) than the size of
> the
> paging file, the system will OOM at allocation time when it's not able to
> back-up the dirty pages to the paging file.
>
>
>
>
> | 2) However, when I moved from 1G to 2G physical memory, the system no
> longer
> | ran into OOM as often as before. With 1G, it could never load all my
> data
> | into memory for processing without seeing OOMs. Apparently adding more
> | physical memory did help.
> |
>
> Yes, now I see why your page file is 1.5GB only, you didn't change it's
> size
> wen you moved from 1GB to 2GB.
>
> | To rephrase my original question, if I can afford any memory size I
> want,
> | which amount would be the max I can benefit from? There must be a
> stopping
> | point somewhere. I thought it was 2G but I could be wrong and hoping
> that
> | I'm wrong because making my code to work with smaller memory amount
> would
> be
> | very costly.
> |
>
> First thing to do is increase your page file size (make it 3GB), and watch
> your paging consumption and the paging rate using perfmon (don't use
> taskman
> for this). If you still incur OOM's, profile your application and check
> your
> allocation patterns carefully, watch for containers like ArrayList, they
> grow exponentially and the are a pain in the a** when they become larger
> than 85KB. Watch your memory fragmentation level when profiling or
> debugging. Memory fragmentation cannot be solved by adding memory.
> Watch your unmanaged memory counters and your unmanaged resources, make
> sure
> you are disposing correctly.
> You should ONLY add RAM when the paging rate is realy too high and becomes
> a
> performance bottleneck. Don't extend above 3GB if you can it's a waste of
> money on 32 bit OS.
>
>
> Willy.
>
>[/color]

[Willy Denoyette [MVP]]

Re: maximum .net process can use

W2K and higher (32 bit OS) supports 16 page files of max. 4GB each. Anyway
you should not set this page file size higher than 3~4GB on a server with
only 2GB RAM.
As I said before a 32 bit windows app. can only use 2GB of Virtual Address
Space, if you think you need this 2GB for one single process, you will
probably need more than 2GB of RAM or you have to accept the possible paging
overhead, but all this is subject to "measure and tune".

Willy.


"Zen" <zen@nononospam .com> wrote in message
news:utP9XWGHGH A.2440@TK2MSFTN GP14.phx.gbl...
| Thanks for you input. Yes, I realized that my page file size was low too,
I
| was surprised that Win2000 didn't automatically change the page file size
to
| take advantage of the max page file size set. Occasionally my page swap
| rate jumped up to 40-50/sec. I'm still wondering about the max size
Windows
| can utilize to serve a resource intensive process, a few things I found:
| 1) On Win2000, max page file is 4G so 4G physical max then.
| 2) On XP, there isn't limitation - I attempted to set it up to 12G and it
| allowed me (I asked me to reboot and I haven't got a chance to do that
| though). Does anyone know the limit? Assume that I have $ to max out XP
| capability to avoid those page swaps.
|
| -thinh
|
|
| "Willy Denoyette [MVP]" <willy.denoyett e@telenet.be> wrote in message
| news:uwMYnSFHGH A.528@TK2MSFTNG P12.phx.gbl...
| >
| > "Zen" <zen@nononospam .com> wrote in message
| > news:uRKqXqEHGH A.2012@TK2MSFTN GP14.phx.gbl...
| > |I used taskmanager that came with window to look at the memory usage
| > info.
| > |
| > | Would it help if I add another G or 2 to the server?
| >
| >
| > No, it won't help you at all, unless you have a high paging rate.
| >
| > Apparently each process
| > | can use only up to 2G max of memory, but other items including the OS
| > itself
| > | would occupy a big junk of total memory leaving less memory for the
| > asp.net.
| > | I don't know the answer the question to that because the following 2
| > items
| > | contradicting to each other:
| > | 1) With virtual memory system and enough harddisk space, the asp.net
| > process
| > | is already virtually working with 2G of memory, so adding physical
| > memory
| > | would only speed up (less swapping with page faults) but woudn't help
| > | reducing OOM exception likelihood. My max page file size on harddisk
is
| > | 1.5G.
| >
| > Did you measure the paging rate? Your paging file is too small anyway,
it
| > should be larger than physical memory, and it should be large enough to
| > hold
| > all 'dirty' datapages of the total of all processes. That means that the
| > OS
| > can never allocate more data segments (as the GC does) than the size of
| > the
| > paging file, the system will OOM at allocation time when it's not able
to
| > back-up the dirty pages to the paging file.
| >
| >
| >
| >
| > | 2) However, when I moved from 1G to 2G physical memory, the system no
| > longer
| > | ran into OOM as often as before. With 1G, it could never load all my
| > data
| > | into memory for processing without seeing OOMs. Apparently adding
more
| > | physical memory did help.
| > |
| >
| > Yes, now I see why your page file is 1.5GB only, you didn't change it's
| > size
| > wen you moved from 1GB to 2GB.
| >
| > | To rephrase my original question, if I can afford any memory size I
| > want,
| > | which amount would be the max I can benefit from? There must be a
| > stopping
| > | point somewhere. I thought it was 2G but I could be wrong and hoping
| > that
| > | I'm wrong because making my code to work with smaller memory amount
| > would
| > be
| > | very costly.
| > |
| >
| > First thing to do is increase your page file size (make it 3GB), and
watch
| > your paging consumption and the paging rate using perfmon (don't use
| > taskman
| > for this). If you still incur OOM's, profile your application and check
| > your
| > allocation patterns carefully, watch for containers like ArrayList, they
| > grow exponentially and the are a pain in the a** when they become larger
| > than 85KB. Watch your memory fragmentation level when profiling or
| > debugging. Memory fragmentation cannot be solved by adding memory.
| > Watch your unmanaged memory counters and your unmanaged resources, make
| > sure
| > you are disposing correctly.
| > You should ONLY add RAM when the paging rate is realy too high and
becomes
| > a
| > performance bottleneck. Don't extend above 3GB if you can it's a waste
of
| > money on 32 bit OS.
| >
| >
| > Willy.
| >
| >
|
|

[Zen]

Re: maximum .net process can use

Each application/process has its own Virtual Address Space of 2G regardless
of how many other processes are running and even on a dual-processor
machine? Thanks!

-zen


"Willy Denoyette [MVP]" <willy.denoyett e@telenet.be> wrote in message
news:Oy5Wk7GHGH A.2680@TK2MSFTN GP09.phx.gbl...[color=blue]
> W2K and higher (32 bit OS) supports 16 page files of max. 4GB each.
> Anyway
> you should not set this page file size higher than 3~4GB on a server with
> only 2GB RAM.
> As I said before a 32 bit windows app. can only use 2GB of Virtual Address
> Space, if you think you need this 2GB for one single process, you will
> probably need more than 2GB of RAM or you have to accept the possible
> paging
> overhead, but all this is subject to "measure and tune".
>
> Willy.
>
>
> "Zen" <zen@nononospam .com> wrote in message
> news:utP9XWGHGH A.2440@TK2MSFTN GP14.phx.gbl...
> | Thanks for you input. Yes, I realized that my page file size was low
> too,
> I
> | was surprised that Win2000 didn't automatically change the page file
> size
> to
> | take advantage of the max page file size set. Occasionally my page swap
> | rate jumped up to 40-50/sec. I'm still wondering about the max size
> Windows
> | can utilize to serve a resource intensive process, a few things I found:
> | 1) On Win2000, max page file is 4G so 4G physical max then.
> | 2) On XP, there isn't limitation - I attempted to set it up to 12G and
> it
> | allowed me (I asked me to reboot and I haven't got a chance to do that
> | though). Does anyone know the limit? Assume that I have $ to max out XP
> | capability to avoid those page swaps.
> |
> | -thinh
> |
> |
> | "Willy Denoyette [MVP]" <willy.denoyett e@telenet.be> wrote in message
> | news:uwMYnSFHGH A.528@TK2MSFTNG P12.phx.gbl...
> | >
> | > "Zen" <zen@nononospam .com> wrote in message
> | > news:uRKqXqEHGH A.2012@TK2MSFTN GP14.phx.gbl...
> | > |I used taskmanager that came with window to look at the memory usage
> | > info.
> | > |
> | > | Would it help if I add another G or 2 to the server?
> | >
> | >
> | > No, it won't help you at all, unless you have a high paging rate.
> | >
> | > Apparently each process
> | > | can use only up to 2G max of memory, but other items including the
> OS
> | > itself
> | > | would occupy a big junk of total memory leaving less memory for the
> | > asp.net.
> | > | I don't know the answer the question to that because the following 2
> | > items
> | > | contradicting to each other:
> | > | 1) With virtual memory system and enough harddisk space, the asp.net
> | > process
> | > | is already virtually working with 2G of memory, so adding physical
> | > memory
> | > | would only speed up (less swapping with page faults) but woudn't
> help
> | > | reducing OOM exception likelihood. My max page file size on
> harddisk
> is
> | > | 1.5G.
> | >
> | > Did you measure the paging rate? Your paging file is too small anyway,
> it
> | > should be larger than physical memory, and it should be large enough
> to
> | > hold
> | > all 'dirty' datapages of the total of all processes. That means that
> the
> | > OS
> | > can never allocate more data segments (as the GC does) than the size
> of
> | > the
> | > paging file, the system will OOM at allocation time when it's not able
> to
> | > back-up the dirty pages to the paging file.
> | >
> | >
> | >
> | >
> | > | 2) However, when I moved from 1G to 2G physical memory, the system
> no
> | > longer
> | > | ran into OOM as often as before. With 1G, it could never load all
> my
> | > data
> | > | into memory for processing without seeing OOMs. Apparently adding
> more
> | > | physical memory did help.
> | > |
> | >
> | > Yes, now I see why your page file is 1.5GB only, you didn't change
> it's
> | > size
> | > wen you moved from 1GB to 2GB.
> | >
> | > | To rephrase my original question, if I can afford any memory size I
> | > want,
> | > | which amount would be the max I can benefit from? There must be a
> | > stopping
> | > | point somewhere. I thought it was 2G but I could be wrong and
> hoping
> | > that
> | > | I'm wrong because making my code to work with smaller memory amount
> | > would
> | > be
> | > | very costly.
> | > |
> | >
> | > First thing to do is increase your page file size (make it 3GB), and
> watch
> | > your paging consumption and the paging rate using perfmon (don't use
> | > taskman
> | > for this). If you still incur OOM's, profile your application and
> check
> | > your
> | > allocation patterns carefully, watch for containers like ArrayList,
> they
> | > grow exponentially and the are a pain in the a** when they become
> larger
> | > than 85KB. Watch your memory fragmentation level when profiling or
> | > debugging. Memory fragmentation cannot be solved by adding memory.
> | > Watch your unmanaged memory counters and your unmanaged resources,
> make
> | > sure
> | > you are disposing correctly.
> | > You should ONLY add RAM when the paging rate is realy too high and
> becomes
> | > a
> | > performance bottleneck. Don't extend above 3GB if you can it's a waste
> of
> | > money on 32 bit OS.
> | >
> | >
> | > Willy.
> | >
> | >
> |
> |
>
>[/color]

[Kevin Frey]

Re: maximum .net process can use

I was referring to the figures presented on the "Performanc e" page of Task
Manager, not in the process page.

"Zen" <zen@nononospam .com> wrote in message
news:OF6wAuEHGH A.1676@TK2MSFTN GP09.phx.gbl...[color=blue]
> There is no "Commit Charge" column to choose from in the Task Manager, I'm
> using Windows2000.
>
>
>
> "Kevin Frey" <kevin_g_frey@h otmail.com> wrote in message
> news:OK6O259GGH A.1132@TK2MSFTN GP10.phx.gbl...[color=green]
>> Firstly in Task Manager look at your "Commit Charge". This is the memory
>> committed to individual processes in total, plus the operating system's
>> memory requirements.
>>
>> See how the "total" compares to the "limit". The limit will effectively
>> be the amount of physical RAM that you have plus your paging space.
>>
>> "Zen" <zen@nononospam .com> wrote in message
>> news:%23IbVdJ9G GHA.2036@TK2MSF TNGP14.phx.gbl. ..[color=darkred]
>>> Hi,
>>>
>>> My production machine has 2G of memory, when aspnet_wp.exe goes up to
>>> about ~1.2G of memory usage, I start get out-of-memory exception. Other
>>> processes don't use as much memory and I added all the peak memory usage
>>> of all the processes (including aspnet_wp.exe), it goes up to no more
>>> than 1.5.
>>>
>>> How is that possible? Would anyone know please help? thanks!
>>>[/color]
>>
>>[/color]
>
>[/color]

[Zen]

Re: maximum .net process can use

ok, found it, and the total is no where near the limit. My limit is 5G,
total about 1.3G, does it tell us anything? By the way, I just ran into
Out-of-memory exceptions again when there were only 1.3G in usage for aspnet
process and even after I increased the page file to 3G min and 4G max. How
could it be possible?



"Kevin Frey" <kevin_g_frey@h otmail.com> wrote in message
news:OvHWfXHHGH A.740@TK2MSFTNG P12.phx.gbl...[color=blue]
>I was referring to the figures presented on the "Performanc e" page of Task
>Manager, not in the process page.
>
> "Zen" <zen@nononospam .com> wrote in message
> news:OF6wAuEHGH A.1676@TK2MSFTN GP09.phx.gbl...[color=green]
>> There is no "Commit Charge" column to choose from in the Task Manager,
>> I'm using Windows2000.
>>
>>
>>
>> "Kevin Frey" <kevin_g_frey@h otmail.com> wrote in message
>> news:OK6O259GGH A.1132@TK2MSFTN GP10.phx.gbl...[color=darkred]
>>> Firstly in Task Manager look at your "Commit Charge". This is the memory
>>> committed to individual processes in total, plus the operating system's
>>> memory requirements.
>>>
>>> See how the "total" compares to the "limit". The limit will effectively
>>> be the amount of physical RAM that you have plus your paging space.
>>>
>>> "Zen" <zen@nononospam .com> wrote in message
>>> news:%23IbVdJ9G GHA.2036@TK2MSF TNGP14.phx.gbl. ..
>>>> Hi,
>>>>
>>>> My production machine has 2G of memory, when aspnet_wp.exe goes up to
>>>> about ~1.2G of memory usage, I start get out-of-memory exception. Other
>>>> processes don't use as much memory and I added all the peak memory
>>>> usage of all the processes (including aspnet_wp.exe), it goes up to no
>>>> more than 1.5.
>>>>
>>>> How is that possible? Would anyone know please help? thanks!
>>>>
>>>
>>>[/color]
>>
>>[/color]
>
>[/color]

[Zen]

Re: maximum .net process can use

Ok, i just ran into the the Out-of-memory exception again. Aspnet process
doesn't seem to be able to use more than 1.3G at the most. Any other idea?
Thanks!


"Willy Denoyette [MVP]" <willy.denoyett e@telenet.be> wrote in message
news:uwMYnSFHGH A.528@TK2MSFTNG P12.phx.gbl...[color=blue]
>
> "Zen" <zen@nononospam .com> wrote in message
> news:uRKqXqEHGH A.2012@TK2MSFTN GP14.phx.gbl...
> |I used taskmanager that came with window to look at the memory usage
> info.
> |
> | Would it help if I add another G or 2 to the server?
>
>
> No, it won't help you at all, unless you have a high paging rate.
>
> Apparently each process
> | can use only up to 2G max of memory, but other items including the OS
> itself
> | would occupy a big junk of total memory leaving less memory for the
> asp.net.
> | I don't know the answer the question to that because the following 2
> items
> | contradicting to each other:
> | 1) With virtual memory system and enough harddisk space, the asp.net
> process
> | is already virtually working with 2G of memory, so adding physical
> memory
> | would only speed up (less swapping with page faults) but woudn't help
> | reducing OOM exception likelihood. My max page file size on harddisk is
> | 1.5G.
>
> Did you measure the paging rate? Your paging file is too small anyway, it
> should be larger than physical memory, and it should be large enough to
> hold
> all 'dirty' datapages of the total of all processes. That means that the
> OS
> can never allocate more data segments (as the GC does) than the size of
> the
> paging file, the system will OOM at allocation time when it's not able to
> back-up the dirty pages to the paging file.
>
>
>
>
> | 2) However, when I moved from 1G to 2G physical memory, the system no
> longer
> | ran into OOM as often as before. With 1G, it could never load all my
> data
> | into memory for processing without seeing OOMs. Apparently adding more
> | physical memory did help.
> |
>
> Yes, now I see why your page file is 1.5GB only, you didn't change it's
> size
> wen you moved from 1GB to 2GB.
>
> | To rephrase my original question, if I can afford any memory size I
> want,
> | which amount would be the max I can benefit from? There must be a
> stopping
> | point somewhere. I thought it was 2G but I could be wrong and hoping
> that
> | I'm wrong because making my code to work with smaller memory amount
> would
> be
> | very costly.
> |
>
> First thing to do is increase your page file size (make it 3GB), and watch
> your paging consumption and the paging rate using perfmon (don't use
> taskman
> for this). If you still incur OOM's, profile your application and check
> your
> allocation patterns carefully, watch for containers like ArrayList, they
> grow exponentially and the are a pain in the a** when they become larger
> than 85KB. Watch your memory fragmentation level when profiling or
> debugging. Memory fragmentation cannot be solved by adding memory.
> Watch your unmanaged memory counters and your unmanaged resources, make
> sure
> you are disposing correctly.
> You should ONLY add RAM when the paging rate is realy too high and becomes
> a
> performance bottleneck. Don't extend above 3GB if you can it's a waste of
> money on 32 bit OS.
>
>
> Willy.
>
>[/color]

[Willy Denoyette [MVP]]

Re: maximum .net process can use

Ok, it looks like you are not carefully reading what I've written or I did
not make myself clear.
Your asp.net worker process can use at most 2GB address space, at a certain
moment in time you have allocated 1.3 GB, and then you try to allocate
another chunk of memory but this fails with an OOM.
The question YOU have to answer is - how large is this chunk of memory you
are trying to allocate, note I say "chunk" of memory because I don't know
what exactly your program is doing, is it creating another object, if yes
what kind of object is it, how large is it? is it allocating a block of heap
memory, how large is this memory block? All I know is that your program is
trying to get memory from it's virtual address space but the space needed is
not available as a contiguous block. The 700 MB free space might (and will)
be so badly fragmented that the memory request for x bytes fails. Question
is HOW LARGE IS X?

To answer all these questions, you'll have to watch your allocation
patterns, you have to look at what your code is doing, what kind of objects
you are creating. The best way to do this is to run a memory profiler
(scitec or the Clrprofiler to name a few) or attach a debugger.
Take a look at your code and check if you aren't using ArrayLists or
DataSet's that can grow exponentially, take care you aren't pinning objects
(PInvoke calls into unmanaged code) etc...

Willy.


"Zen" <zen@nononospam .com> wrote in message
news:uuX6PeHHGH A.3624@TK2MSFTN GP09.phx.gbl...
| Ok, i just ran into the the Out-of-memory exception again. Aspnet process
| doesn't seem to be able to use more than 1.3G at the most. Any other idea?
| Thanks!
|
|
| "Willy Denoyette [MVP]" <willy.denoyett e@telenet.be> wrote in message
| news:uwMYnSFHGH A.528@TK2MSFTNG P12.phx.gbl...
| >
| > "Zen" <zen@nononospam .com> wrote in message
| > news:uRKqXqEHGH A.2012@TK2MSFTN GP14.phx.gbl...
| > |I used taskmanager that came with window to look at the memory usage
| > info.
| > |
| > | Would it help if I add another G or 2 to the server?
| >
| >
| > No, it won't help you at all, unless you have a high paging rate.
| >
| > Apparently each process
| > | can use only up to 2G max of memory, but other items including the OS
| > itself
| > | would occupy a big junk of total memory leaving less memory for the
| > asp.net.
| > | I don't know the answer the question to that because the following 2
| > items
| > | contradicting to each other:
| > | 1) With virtual memory system and enough harddisk space, the asp.net
| > process
| > | is already virtually working with 2G of memory, so adding physical
| > memory
| > | would only speed up (less swapping with page faults) but woudn't help
| > | reducing OOM exception likelihood. My max page file size on harddisk
is
| > | 1.5G.
| >
| > Did you measure the paging rate? Your paging file is too small anyway,
it
| > should be larger than physical memory, and it should be large enough to
| > hold
| > all 'dirty' datapages of the total of all processes. That means that the
| > OS
| > can never allocate more data segments (as the GC does) than the size of
| > the
| > paging file, the system will OOM at allocation time when it's not able
to
| > back-up the dirty pages to the paging file.
| >
| >
| >
| >
| > | 2) However, when I moved from 1G to 2G physical memory, the system no
| > longer
| > | ran into OOM as often as before. With 1G, it could never load all my
| > data
| > | into memory for processing without seeing OOMs. Apparently adding
more
| > | physical memory did help.
| > |
| >
| > Yes, now I see why your page file is 1.5GB only, you didn't change it's
| > size
| > wen you moved from 1GB to 2GB.
| >
| > | To rephrase my original question, if I can afford any memory size I
| > want,
| > | which amount would be the max I can benefit from? There must be a
| > stopping
| > | point somewhere. I thought it was 2G but I could be wrong and hoping
| > that
| > | I'm wrong because making my code to work with smaller memory amount
| > would
| > be
| > | very costly.
| > |
| >
| > First thing to do is increase your page file size (make it 3GB), and
watch
| > your paging consumption and the paging rate using perfmon (don't use
| > taskman
| > for this). If you still incur OOM's, profile your application and check
| > your
| > allocation patterns carefully, watch for containers like ArrayList, they
| > grow exponentially and the are a pain in the a** when they become larger
| > than 85KB. Watch your memory fragmentation level when profiling or
| > debugging. Memory fragmentation cannot be solved by adding memory.
| > Watch your unmanaged memory counters and your unmanaged resources, make
| > sure
| > you are disposing correctly.
| > You should ONLY add RAM when the paging rate is realy too high and
becomes
| > a
| > performance bottleneck. Don't extend above 3GB if you can it's a waste
of
| > money on 32 bit OS.
| >
| >
| > Willy.
| >
| >
|
|

[Neyah]

RE: maximum .net process can use

See this info (it's long)

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnaspp/html/monitor_perf.asp

A quote:
First, the likelihood of experiencing an OutOfMemoryExce ption begins to
increase dramatically when "Process\Virtua l Bytes" is within 600 MB of the
virtual address space limit (generally 2 GB), and secondly, tests have shown
that "Process\Virtua l Bytes" is often larger than "Process\Privat e Bytes" by
no more than 600 MB. This difference is due in part to the MEM_RESERVE
regions maintained by the GC, allowing it to quickly commit more memory when
needed. Taken together this implies that when "Process\Privat e Bytes" exceeds
800 MB, the likelihood of experiencing an OutOfMemoryExce ption increases.

Long story short, asp.net 1.1 with has a high likelyhood of experiencing
OutOfMemoryExce ptions once the aspnet_wp.exe process exceeds 800 MB due to
address space limitations and additional reserve by the GC.

The below applies to IIS 5.0. For IIS 6.0, you would set this up from IIS
Manager Properties -> Recycling -> Maximum used memory. But one of your
other posts indicated you were using Win 2k server, so you don't have IIS 6.0.

The memorylimit defaults to 60% of available RAM, which is why you didn't
have any issues while you only had 1 GB. Now that you have 2 GB,
aspnet_wp.exe won't recycle until it reaches 1.2 GB, which is over the 800 MB
problem point. You'll want to set the memoryLimit in the your machine.config
setting for processModel to a value that doesn't exceed 800 MB. For a 2 GB
machine, 39% will do the trick.

If you have enough RAM to where you have included the /3GB switch to
boot.ini, you can use a value that calculates out to 1,800 MB instead.

I'd also recommend lowering the timout and idelTimout values from Infinite,
to a lower value. This will allow the aspnet_wp.exe process to actually
recycle.

"Zen" wrote:
[color=blue]
> Hi,
>
> My production machine has 2G of memory, when aspnet_wp.exe goes up to about
> ~1.2G of memory usage, I start get out-of-memory exception. Other processes
> don't use as much memory and I added all the peak memory usage of all the
> processes (including aspnet_wp.exe), it goes up to no more than 1.5.
>
> How is that possible? Would anyone know please help? thanks!
>
>
>[/color]

[Willy Denoyette [MVP]]

Re: maximum .net process can use

Note that this whole story is based on version 1.1 of the framework, this
version and version 1.0 had some problems with GC heap fragmentation
especially the LOH. Most of these problems (but not all) were solved with
SP1 of the framework and I suppose the OP runs v1.1 SP1. V2's memory manager
has even better handling of fragmentation built-in, but fragmentation can
never be avoided completely without the help of the application designer.
Note also that the OP suffered from OOM before the threshold was reached and
it was even worse when he ran on 1GB.
Now there are two things you can do:
1. lower the threshold and let the process recycle before an OOM can posible
occur.
2. Run a profiler and inspect (and adapt) your memory allocation profile.

1. IMO is not a valid option, it's plain ridiculous to have 2GB memory in a
server from which only < 600 MB can be used (remember the OP had OOM's with
1GB and 60% threshold).
2. This is what the OP must do first, it makes no sense to recycle a process
to recover from a bug or an application design mistake.

Willy.

"Neyah" <Neyah@discussi ons.microsoft.c om> wrote in message
news:2A019F44-5E44-4435-8374-552A157D7B7D@mi crosoft.com...
| See this info (it's long)
|

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnaspp/html/monitor_perf.asp

|
| A quote:
| First, the likelihood of experiencing an OutOfMemoryExce ption begins to
| increase dramatically when "Process\Virtua l Bytes" is within 600 MB of the
| virtual address space limit (generally 2 GB), and secondly, tests have
shown
| that "Process\Virtua l Bytes" is often larger than "Process\Privat e Bytes"
by
| no more than 600 MB. This difference is due in part to the MEM_RESERVE
| regions maintained by the GC, allowing it to quickly commit more memory
when
| needed. Taken together this implies that when "Process\Privat e Bytes"
exceeds
| 800 MB, the likelihood of experiencing an OutOfMemoryExce ption increases.
|
| Long story short, asp.net 1.1 with has a high likelyhood of experiencing
| OutOfMemoryExce ptions once the aspnet_wp.exe process exceeds 800 MB due to
| address space limitations and additional reserve by the GC.
|
| The below applies to IIS 5.0. For IIS 6.0, you would set this up from IIS
| Manager Properties -> Recycling -> Maximum used memory. But one of your
| other posts indicated you were using Win 2k server, so you don't have IIS
6.0.
|
| The memorylimit defaults to 60% of available RAM, which is why you didn't
| have any issues while you only had 1 GB. Now that you have 2 GB,
| aspnet_wp.exe won't recycle until it reaches 1.2 GB, which is over the 800
MB
| problem point. You'll want to set the memoryLimit in the your
machine.config
| setting for processModel to a value that doesn't exceed 800 MB. For a 2
GB
| machine, 39% will do the trick.
|
| If you have enough RAM to where you have included the /3GB switch to
| boot.ini, you can use a value that calculates out to 1,800 MB instead.
|
| I'd also recommend lowering the timout and idelTimout values from
Infinite,
| to a lower value. This will allow the aspnet_wp.exe process to actually
| recycle.
|
| "Zen" wrote:
|
| > Hi,
| >
| > My production machine has 2G of memory, when aspnet_wp.exe goes up to
about
| > ~1.2G of memory usage, I start get out-of-memory exception. Other
processes
| > don't use as much memory and I added all the peak memory usage of all
the
| > processes (including aspnet_wp.exe), it goes up to no more than 1.5.
| >
| > How is that possible? Would anyone know please help? thanks!
| >
| >
| >