Maximum Size of Byte Array

[Gerrit]

Hi all,

I'm getting an OutOfMemoryExce ption when I initialize a byte array in
C# like this:

Byte[] test = new Byte[420000000];

I'm using ASP.NET 2.0. In ASP.Net 1.1 it works fine. So what am I doing
wrong?

What is the maximum length of a byte array?

Can somebody help me? Thanks in advance!

--
Gerrit Horeis

Software Developer
CI-Gate Development & Consulting GmbH

http://www.ci-gate.de

http://www.xira.de

http://www.bitbauer.de

[Jon Skeet [C# MVP]]

Re: Maximum Size of Byte Array

Gerrit <Gerrit.Horeis@ web.dewrote:

I'm getting an OutOfMemoryExce ption when I initialize a byte array in
C# like this:
>
Byte[] test = new Byte[420000000];
>
I'm using ASP.NET 2.0. In ASP.Net 1.1 it works fine. So what am I doing
wrong?
>
What is the maximum length of a byte array?
>
Can somebody help me? Thanks in advance!

My guess is that ASP.NET 2.0 has some extra constraints in terms of how
much memory your app is allowed to use, to prevent wayward apps from
killing the machine. If I may say so, that's a fairly extreme use of
memory...

I can't say I know much about ASP.NET configuration - perhaps someone
in the ASP.NET group could help you there - but my gut feeling is
that's where the problem is.

--
Jon Skeet - <skeet@pobox.co m>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

[Gerrit]

Re: Maximum Size of Byte Array

Thank you very much!

The strange on this behavior is that it sometimes occurs and sometimes
not.
But I'll try my luck in a ASP.NET group.

Thank you so far!

--
Gerrit Horeis

Software Developer
CI-Gate Development & Consulting GmbH

http://www.ci-gate.de

http://www.xira.de

http://www.bitbauer.de

Jon schrieb:

Gerrit <Gerrit.Horeis@ web.dewrote:

I'm getting an OutOfMemoryExce ption when I initialize a byte array in
C# like this:

Byte[] test = new Byte[420000000];

I'm using ASP.NET 2.0. In ASP.Net 1.1 it works fine. So what am I doing
wrong?

What is the maximum length of a byte array?

Can somebody help me? Thanks in advance!

>
My guess is that ASP.NET 2.0 has some extra constraints in terms of how
much memory your app is allowed to use, to prevent wayward apps from
killing the machine. If I may say so, that's a fairly extreme use of
memory...
>
I can't say I know much about ASP.NET configuration - perhaps someone
in the ASP.NET group could help you there - but my gut feeling is
that's where the problem is.
>
--
Jon Skeet - <skeet@pobox.co m>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

[Thomas Scheidegger [MVP]]

Re: Maximum Size of Byte Array

The strange on this behavior is that it sometimes occurs and sometimes not.


I guess it could also be some fragmentation of heaps / virtual address space.

=Get a 64-Bit system ;-) {but even 64-Bit .NET 2.0 has some 32-Bit limitations }

http://blogs.msdn.com/tess/archive/2005/11/25/496898.aspx

http://msdn.microsoft.com/library/en-us/dnaspp/html/monitor_perf.asp

--
Thomas Scheidegger - MVP .NET - 'NETMaster'
http://www.cetus-links.org/oo_dotnet.html - http://dnetmaster.net/

[Willy Denoyette [MVP]]

Re: Maximum Size of Byte Array


"Gerrit" <Gerrit.Horeis@ web.dewrote in message
news:1159271280 .382689.119240@ h48g2000cwc.goo glegroups.com.. .
| Hi all,
|
| I'm getting an OutOfMemoryExce ption when I initialize a byte array in
| C# like this:
|
| Byte[] test = new Byte[420000000];
|
| I'm using ASP.NET 2.0. In ASP.Net 1.1 it works fine. So what am I doing
| wrong?
|
| What is the maximum length of a byte array?
|
| Can somebody help me? Thanks in advance!
|
| --
| Gerrit Horeis
|
| Software Developer
| CI-Gate Development & Consulting GmbH
| http://www.ci-gate.de
| http://www.xira.de
| http://www.bitbauer.de
|

You should never assume that any portion of memory is available, even a
simple object allocation may fail. Now, as for your array of bytes,
theoretically the maximum size is ~2 billion bytes, but you will never be
able to allocate that size from a 32 bit process. Same goes for your 420
Mbytes array, allocating this from a 32 bit process will certainly lead to
OOM failures, as a result of heap fragmentation.
There is no guarantee that such an amount of contiguous memory space will be
available, certainly not when allocated in long running applications like
services (asp.net). For asp.net there is also a configuration parameter
which limits the total size of the process memory to a certain percentages
of the available virtual memory, while this is something to watch for, the
only solution to your problem, redesign your application in order to reduce
your memory consumption (this is a memory hog which doesn't scale at all),
or move to a 64 bit OS.

Willy.

[Ignacio Machin \( .NET/ C# MVP \)]

Re: Maximum Size of Byte Array

Hi,


"Gerrit" <Gerrit.Horeis@ web.dewrote in message
news:1159271280 .382689.119240@ h48g2000cwc.goo glegroups.com.. .

Hi all,
>
I'm getting an OutOfMemoryExce ption when I initialize a byte array in
C# like this:
>
Byte[] test = new Byte[420000000];
>
I'm using ASP.NET 2.0. In ASP.Net 1.1 it works fine. So what am I doing
wrong?
>
What is the maximum length of a byte array?

Why you need such a big array in the first place?

I bet it depends of either (or both) IIS and the framework

--
--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation

[Willy Denoyette [MVP]]

Re: Maximum Size of Byte Array


"Ignacio Machin ( .NET/ C# MVP )" <ignacio.mach in AT dot.state.fl.us wrote
in message news:%23XUUSJX4 GHA.4740@TK2MSF TNGP03.phx.gbl. ..
| Hi,
|
|
| "Gerrit" <Gerrit.Horeis@ web.dewrote in message
| news:1159271280 .382689.119240@ h48g2000cwc.goo glegroups.com.. .
| Hi all,
| >
| I'm getting an OutOfMemoryExce ption when I initialize a byte array in
| C# like this:
| >
| Byte[] test = new Byte[420000000];
| >
| I'm using ASP.NET 2.0. In ASP.Net 1.1 it works fine. So what am I doing
| wrong?
| >
| What is the maximum length of a byte array?
|
|
| Why you need such a big array in the first place?
|
| I bet it depends of either (or both) IIS and the framework
|


Hmm... IIS has nothing to do with this, ASP.NET runs in it's own process,
the framework only limit's the byte array size (just like any other object)
to ~2 GByte.
It's the way you allocate large contiguous blocks of memory from your
available virtual address space on a 32 bit OS which predicts the amount of
memory you can allocate. No single process, unless very carefully designed
and tested, will run for long before it encounters OOM exceptions when you
start allocating large objects or huge array's at wild. That's why asp.net
(the hosting process) provides the possibility to restart an AD and or the
worker process when a configurable memory threshold has been reached.

Willy.

[Gerrit]

Re: Maximum Size of Byte Array

Thanks for your answer!

I want to upload an image to a web application and therefore i need to
allocate this memory to fill the byte array with image data. Then I
need to do a resizing operation.

Ignacio Machin ( .NET/ C# MVP ) schrieb:

Hi,
>
>
"Gerrit" <Gerrit.Horeis@ web.dewrote in message
news:1159271280 .382689.119240@ h48g2000cwc.goo glegroups.com.. .

Hi all,

I'm getting an OutOfMemoryExce ption when I initialize a byte array in
C# like this:

Byte[] test = new Byte[420000000];

I'm using ASP.NET 2.0. In ASP.Net 1.1 it works fine. So what am I doing
wrong?

What is the maximum length of a byte array?

>
>
Why you need such a big array in the first place?
>
I bet it depends of either (or both) IIS and the framework
>
--
--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation

[Gerrit]

Re: Maximum Size of Byte Array

I think I'll try your advice at home where I have a 64Bit system but
with WinXP 32Bit OS installed. Do I also need to install a 64Bit OS to
try that ? (would be a task for the weekend ;-))


Thomas Scheidegger [MVP] schrieb:

The strange on this behavior is that it sometimes occurs and sometimes not.

>
>
I guess it could also be some fragmentation of heaps / virtual address space.
>
=Get a 64-Bit system ;-) {but even 64-Bit .NET 2.0 has some 32-Bit limitations }
>

http://blogs.msdn.com/tess/archive/2005/11/25/496898.aspx

>

http://msdn.microsoft.com/library/en-us/dnaspp/html/monitor_perf.asp

>
>
--
Thomas Scheidegger - MVP .NET - 'NETMaster'
http://www.cetus-links.org/oo_dotnet.html - http://dnetmaster.net/

[Thomas Scheidegger [MVP]]

Re: Maximum Size of Byte Array

>I think I'll try your advice at home where I have a 64Bit system but

with WinXP 32Bit OS installed. Do I also need to install a 64Bit OS to
try that ? (would be a task for the weekend ;-))

yes both, 64-Bit Windows (XP/2003/[Vista])
and the 64-Bit .NET Runtime.
For x64:

http://www.microsoft.com/downloads/details.aspx?familyid=B44A0000-ACF8-4FA1-AFFB-40E78D788B00

And compile for 'AnyCpu' target (or x64).

And maybe switching ASP.NET to 64-Bit

http://support.microsoft.com/kb/894435/en-us

--
Thomas Scheidegger - MVP .NET - 'NETMaster'
http://www.cetus-links.org/oo_dotnet.html - http://dnetmaster.net/

[Thomas Scheidegger [MVP]]

Re: Maximum Size of Byte Array

and the 64-Bit .NET Runtime.


for array limitations

http://blogs.msdn.com/joshwil/archive/2005/08/10/450202.aspx

--
Thomas Scheidegger - MVP .NET - 'NETMaster'
http://www.cetus-links.org/oo_dotnet.html - http://dnetmaster.net/

[Jon Skeet [C# MVP]]

Re: Maximum Size of Byte Array

Gerrit <Gerrit.Horeis@ web.dewrote:

I want to upload an image to a web application and therefore i need to
allocate this memory to fill the byte array with image data. Then I
need to do a resizing operation.

Is your image really 420MB large? That's really pretty massive. Can you
not do it in pieces? If not, perhaps you could launch a different
process to do the loading and resizing. That process could write the
result to disk, and then you could stream it to the web application.

--
Jon Skeet - <skeet@pobox.co m>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

[Gerrit]

Re: Maximum Size of Byte Array

Hi Jon,

yes this pictures are really up to 1GB. My method to shrink the images
works fine until a (decompressed) FileSize of 350 MB is reached.

If I do it in pieces, I also need to decompress the Picture on harddisk
as a bmp file and then read it into memory in small pieces. But to
decompress such a large picture with an .Net service or something i
will have to load it into memory anyway, which would cause a OOM
exception.

Do I have to use unmanaged code? Or is there a Windows API or anything
else I could use?

--
Gerrit Horeis

Software Developer
CI-Gate Development & Consulting GmbH

http://www.ci-gate.de

http://www.xira.de

http://www.bitbauer.de

Jon schrieb:

Gerrit <Gerrit.Horeis@ web.dewrote:

I want to upload an image to a web application and therefore i need to
allocate this memory to fill the byte array with image data. Then I
need to do a resizing operation.

>
Is your image really 420MB large? That's really pretty massive. Can you
not do it in pieces? If not, perhaps you could launch a different
process to do the loading and resizing. That process could write the
result to disk, and then you could stream it to the web application.
>
--
Jon Skeet - <skeet@pobox.co m>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

[Willy Denoyette [MVP]]

Re: Maximum Size of Byte Array

As been said before in this thread, you won't be able to handle such huge
arrays on a 32 bit platform, not using unmanaged code or anything else.
You can't rely on the fact that such a huge free chunk of (CONTIGUOUS)
memory will ever be available, and if it's available at some point in time,
chances are that it's no longer available some time later as a result of
heap fragmentation.
A 32 bit process has a virtual memory space of max. ~GAB (or ~GAB for /GET
RAM tuned system), this space is shared by things like code modules, data
files, private heaps, process heaps the thread stacks etc...
.. After process creation and loading the initial modules (DLL's) this VIM
space is already fragmented, such that only ~1.GAB is available in one large
chunk (this figure is for the asp.net worker process).
But there is something else what scares me, when you are talking about
images and bitmaps in asp.net, does it mean that you are using
System.Drawing classes? If that's the case you should keep in mind that this
namespace is not supported (for good reasons) in asp nor in any other
service based application.

Willy.


"Gerrit" <Gerrit.Horeis@ web.dewrote in message
news:1159348938 .778802.250860@ b28g2000cwb.goo glegroups.com.. .
| Hi Jon,
|
| yes this pictures are really up to 1GB. My method to shrink the images
| works fine until a (decompressed) FileSize of 350 MB is reached.
|
| If I do it in pieces, I also need to decompress the Picture on harddisk
| as a bmp file and then read it into memory in small pieces. But to
| decompress such a large picture with an .Net service or something i
| will have to load it into memory anyway, which would cause a OOM
| exception.
|
| Do I have to use unmanaged code? Or is there a Windows API or anything
| else I could use?
|
| --
| Gerrit Horeis
|
| Software Developer
| CI-Gate Development & Consulting GmbH
| http://www.ci-gate.de
| http://www.xira.de
| http://www.bitbauer.de
|
|
| Jon schrieb:
|
| Gerrit <Gerrit.Horeis@ web.dewrote:
| I want to upload an image to a web application and therefore i need to
| allocate this memory to fill the byte array with image data. Then I
| need to do a resizing operation.
| >
| Is your image really 420MB large? That's really pretty massive. Can you
| not do it in pieces? If not, perhaps you could launch a different
| process to do the loading and resizing. That process could write the
| result to disk, and then you could stream it to the web application.
| >
| --
| Jon Skeet - <skeet@pobox.co m>
| http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
| If replying to the group, please do not mail me too
|

[Ignacio Machin \( .NET/ C# MVP \)]

Re: Maximum Size of Byte Array

Hi,

You will not be able to handle such a big image in memory. You would have to
devise a way to use it in parts.
Also select carefully your tools, I don't think System.Drawing can handle
such a big image, you would be better using DirectX or even better image
manipulation code written in unmanaged code.


--
--
Ignacio Machin,
ignacio.machin AT dot.state.fl.us
Florida Department Of Transportation

"Gerrit" <Gerrit.Horeis@ web.dewrote in message
news:1159348938 .778802.250860@ b28g2000cwb.goo glegroups.com.. .

Hi Jon,
>
yes this pictures are really up to 1GB. My method to shrink the images
works fine until a (decompressed) FileSize of 350 MB is reached.
>
If I do it in pieces, I also need to decompress the Picture on harddisk
as a bmp file and then read it into memory in small pieces. But to
decompress such a large picture with an .Net service or something i
will have to load it into memory anyway, which would cause a OOM
exception.
>
Do I have to use unmanaged code? Or is there a Windows API or anything
else I could use?
>
--
Gerrit Horeis
>
Software Developer
CI-Gate Development & Consulting GmbH

http://www.ci-gate.de

http://www.xira.de

http://www.bitbauer.de

>
>
Jon schrieb:
>

>Gerrit <Gerrit.Horeis@ web.dewrote:

I want to upload an image to a web application and therefore i need to
allocate this memory to fill the byte array with image data. Then I
need to do a resizing operation.

>>
>Is your image really 420MB large? That's really pretty massive. Can you
>not do it in pieces? If not, perhaps you could launch a different
>process to do the loading and resizing. That process could write the
>result to disk, and then you could stream it to the web application.
>>
>--
>Jon Skeet - <skeet@pobox.co m>
>http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
>If replying to the group, please do not mail me too

>