How does "new" work in a loop?

[Tony Sinclair]

I'm just learning C#. I'm writing a program (using Visual C# 2005 on
WinXP) to combine several files into one (HKSplit is a popular
freeware program that does this, but it requires all input and output
to be within one directory, and I want to be able to combine files
from different directories into another directory of my choice).

My program seems to work fine, but I'm wondering about this loop:


for (int i = 0; i < numFiles; i++)
{
// read next input file

FileStream fs = new FileStream(file Names[i],
FileMode.Open, FileAccess.Read , FileShare.Read) ;
Byte[] inputBuffer = new Byte[fs.Length];

fs.Read(inputBu ffer, 0, (int)fs.Length) ;
fs.Close();

//append to output stream previously opened as fsOut

fsOut.Write(inp utBuffer, 0, (int) inputBuffer.Len gth);
progBar.Value++ ;
} // for int i

As you can see, the objects fs and inputBuffer are both created as
"new" each time through the loop, which could be many times. I didn't
think this would work; I just tried it to see what kind of error
message I would get, and I was surprised when it ran. Every test run
has produced perfect results.

So what is happening here? Is the memory being reused, or am I piling
up objects on the heap that will only go away when my program ends, or
am I creating a huge memory leak?

I can see that fs might go away after fs.Close(), but I don't
understand why I'm allowed to recreate the byte array over and over,
without ever disposing of it. I have verifed with the debugger that
the array has a different size each time the input file size changes,
so it really is being reallocated each time through the loop, rather
than just being reused. I've tried to find explanations of how "new"
works in a loop, but I haven't been able to so far. Any help,
including pointers to the VS docs or a popular book on C#, would be
appreciated.

[Peter Rilling]

Re: How does &quot;new&qu ot; work in a loop?

Well, "new" always create a new instance of the object. The object was
assigned to the variable during the previous cycle will be flagged so the
garbage collector can release memory. Now, since the GC does not clear
memory immediately, there is a period of time when the old object stays in
memory so there is the change that you can use too much memory before the GC
is invoked, but not a great chance. It is not something that I would worry
about unless you plan to join files where each part is a gig in size. :)




"Tony Sinclair" <no@spam.comwro te in message
news:jvqqa2h237 ekpg60jjn3ahc7i 17lum8787@4ax.c om...

I'm just learning C#. I'm writing a program (using Visual C# 2005 on
WinXP) to combine several files into one (HKSplit is a popular
freeware program that does this, but it requires all input and output
to be within one directory, and I want to be able to combine files
from different directories into another directory of my choice).
>
My program seems to work fine, but I'm wondering about this loop:
>
>
for (int i = 0; i < numFiles; i++)
{
// read next input file
>
FileStream fs = new FileStream(file Names[i],
FileMode.Open, FileAccess.Read , FileShare.Read) ;
Byte[] inputBuffer = new Byte[fs.Length];
>
fs.Read(inputBu ffer, 0, (int)fs.Length) ;
fs.Close();
>
//append to output stream previously opened as fsOut
>
fsOut.Write(inp utBuffer, 0, (int) inputBuffer.Len gth);
progBar.Value++ ;
} // for int i
>
As you can see, the objects fs and inputBuffer are both created as
"new" each time through the loop, which could be many times. I didn't
think this would work; I just tried it to see what kind of error
message I would get, and I was surprised when it ran. Every test run
has produced perfect results.
>
So what is happening here? Is the memory being reused, or am I piling
up objects on the heap that will only go away when my program ends, or
am I creating a huge memory leak?
>
I can see that fs might go away after fs.Close(), but I don't
understand why I'm allowed to recreate the byte array over and over,
without ever disposing of it. I have verifed with the debugger that
the array has a different size each time the input file size changes,
so it really is being reallocated each time through the loop, rather
than just being reused. I've tried to find explanations of how "new"
works in a loop, but I haven't been able to so far. Any help,
including pointers to the VS docs or a popular book on C#, would be
appreciated.

[John J. Hughes II]

Re: How does &quot;new&qu ot; work in a loop?

The memory is allocated during each loop and in concept released when the
loop ends. In reality the memory is marked to be released by GC (garbage
collection) as some future point in time. I have found that if you were to
do this loop only a few times and the program was idle the memory would be
freed but if this loop involves a lot of files and very little idle time is
returned to the system you will run out of memory.

That being said I would at the very least place fs=null and inputbuffer =
null at the end of the loop.

A better solution for fs would be the using statment which would force GC
and return the memory.

using(FileStrea m fs = new FileStream(file Names[i],FileMode.Open,
FileAccess.Read , FileShare.Read) )
{
Byte[] inputBuffer = new Byte[fs.Length];

/// do something with fs

inputBuffer = null;
}

Regards,
John

"Tony Sinclair" <no@spam.comwro te in message
news:jvqqa2h237 ekpg60jjn3ahc7i 17lum8787@4ax.c om...

I'm just learning C#. I'm writing a program (using Visual C# 2005 on
WinXP) to combine several files into one (HKSplit is a popular
freeware program that does this, but it requires all input and output
to be within one directory, and I want to be able to combine files
from different directories into another directory of my choice).
>
My program seems to work fine, but I'm wondering about this loop:
>
>
for (int i = 0; i < numFiles; i++)
{
// read next input file
>
FileStream fs = new FileStream(file Names[i],
FileMode.Open, FileAccess.Read , FileShare.Read) ;
Byte[] inputBuffer = new Byte[fs.Length];
>
fs.Read(inputBu ffer, 0, (int)fs.Length) ;
fs.Close();
>
//append to output stream previously opened as fsOut
>
fsOut.Write(inp utBuffer, 0, (int) inputBuffer.Len gth);
progBar.Value++ ;
} // for int i
>
As you can see, the objects fs and inputBuffer are both created as
"new" each time through the loop, which could be many times. I didn't
think this would work; I just tried it to see what kind of error
message I would get, and I was surprised when it ran. Every test run
has produced perfect results.
>
So what is happening here? Is the memory being reused, or am I piling
up objects on the heap that will only go away when my program ends, or
am I creating a huge memory leak?
>
I can see that fs might go away after fs.Close(), but I don't
understand why I'm allowed to recreate the byte array over and over,
without ever disposing of it. I have verifed with the debugger that
the array has a different size each time the input file size changes,
so it really is being reallocated each time through the loop, rather
than just being reused. I've tried to find explanations of how "new"
works in a loop, but I haven't been able to so far. Any help,
including pointers to the VS docs or a popular book on C#, would be
appreciated.

[Matt]

Re: How does &quot;new&qu ot; work in a loop?


Tony Sinclair wrote:

I'm just learning C#. I'm writing a program (using Visual C# 2005 on
WinXP) to combine several files into one (HKSplit is a popular
freeware program that does this, but it requires all input and output
to be within one directory, and I want to be able to combine files
from different directories into another directory of my choice).
>
My program seems to work fine, but I'm wondering about this loop:
>
>
for (int i = 0; i < numFiles; i++)
{
// read next input file
>
FileStream fs = new FileStream(file Names[i],
FileMode.Open, FileAccess.Read , FileShare.Read) ;
Byte[] inputBuffer = new Byte[fs.Length];
>
fs.Read(inputBu ffer, 0, (int)fs.Length) ;
fs.Close();
>
//append to output stream previously opened as fsOut
>
fsOut.Write(inp utBuffer, 0, (int) inputBuffer.Len gth);
progBar.Value++ ;
} // for int i
>
As you can see, the objects fs and inputBuffer are both created as
"new" each time through the loop, which could be many times. I didn't
think this would work; I just tried it to see what kind of error
message I would get, and I was surprised when it ran. Every test run
has produced perfect results.
>
So what is happening here? Is the memory being reused, or am I piling
up objects on the heap that will only go away when my program ends, or
am I creating a huge memory leak?

Unlikely that you are creating a memory leak. C# uses garbage
collection.
When the object goes out of scope (in your case, the } marked // for
int i)
the object is destroyed. The next time through the loop, a new one is
created.

Matt

[Jon Skeet [C# MVP]]

Re: How does &quot;new&qu ot; work in a loop?

John J. Hughes II <no@invalid.com wrote:

The memory is allocated during each loop and in concept released when the
loop ends. In reality the memory is marked to be released by GC (garbage
collection) as some future point in time. I have found that if you were to
do this loop only a few times and the program was idle the memory would be
freed but if this loop involves a lot of files and very little idle time is
returned to the system you will run out of memory.
>
That being said I would at the very least place fs=null and inputbuffer =
null at the end of the loop.

Why? It serves no purpose - and code which serves no purpose is just
distracting, IMO.

A better solution for fs would be the using statment which would force GC
and return the memory.

It wouldn't return the memory - but it *would* close/dispose the stream
in all situations, whether or not there's an exception.
Closing/disposing the stream doesn't return any memory, but it releases
the handle on the file.

--
Jon Skeet - <skeet@pobox.co m>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

[Jon Skeet [C# MVP]]

Re: How does &quot;new&qu ot; work in a loop?

Matt <matttelles@spr ynet.comwrote:

<snip>

Unlikely that you are creating a memory leak. C# uses garbage
collection. When the object goes out of scope (in your case, the }
marked // for int i) the object is destroyed. The next time through
the loop, a new one is created.

The object is *not* destroyed when it reaches the end of the scope.
..NET does not have deterministic garbage collection. Instead, the
object's memory will be released *at some point* after it is last used.

In fact, this could be before the end of the scope - the GC could kick
in before progBar.Value++ and free both fs and inputBuffer.

--
Jon Skeet - <skeet@pobox.co m>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

[Jon Skeet [C# MVP]]

Re: How does &quot;new&qu ot; work in a loop?

Peter Rilling <peter@nospam.r illing.netwrote :

Well, "new" always create a new instance of the object. The object was
assigned to the variable during the previous cycle will be flagged so the
garbage collector can release memory.

Not quite - the "old" object isn't marked in any way (which would
basically be like reference counting). Instead, every time the GC kicks
in, all the "live" objects in the system are marked, and after that's
finished anything which *isn't* marked can be destroyed (or finalized,
if it has a finalizer).

--
Jon Skeet - <skeet@pobox.co m>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

[Barry Kelly]

Re: How does &quot;new&qu ot; work in a loop?

"Matt" <matttelles@spr ynet.comwrote:

So what is happening here? Is the memory being reused, or am I piling
up objects on the heap that will only go away when my program ends, or
am I creating a huge memory leak?

>
Unlikely that you are creating a memory leak. C# uses garbage
collection. When the object goes out of scope (in your case, the }
marked // for int i) the object is destroyed. The next time through
the loop, a new one is created.

It isn't freed immediately on every loop, or when it goes out of scope,
unless it's a value type (struct in C#). The GC kicks in on an as-needed
basis: memory pressure, repeated allocations, etc. That's one thing, and
is easy.

The other half of the story is resources. The GC isn't a resource
manager, so you shouldn't rely on it to manage things like file handles,
since it won't release them in a timely manner. That's why it's
important to use "using" with FileStream and other classes which
implement IDispose.

-- Barry

--

http://barrkel.blogspot.com/

[Barry Kelly]

Re: How does &quot;new&qu ot; work in a loop?

Barry Kelly <barry.j.kelly@ gmail.comwrote:

important to use "using" with FileStream and other classes which
implement IDispose.

IDisposable rather

-- Barry

--

http://barrkel.blogspot.com/

[Göran Andersson]

Re: How does &quot;new&qu ot; work in a loop?

Jon Skeet [C# MVP] wrote:

John J. Hughes II <no@invalid.com wrote:

>The memory is allocated during each loop and in concept released when the
>loop ends. In reality the memory is marked to be released by GC (garbage
>collection) as some future point in time. I have found that if you were to
>do this loop only a few times and the program was idle the memory would be
>freed but if this loop involves a lot of files and very little idle time is
>returned to the system you will run out of memory.
>>
>That being said I would at the very least place fs=null and inputbuffer =
>null at the end of the loop.

>
Why? It serves no purpose - and code which serves no purpose is just
distracting, IMO.

Well, it actually serves one purpose, at least occationally.

If the first generation of the heap is full when the buffer is going to
be allocated, a garbage collection kicks in to free some memory. If you
have removed the reference to the previous buffer, it can be collected,
otherwise not.

For that purpose, you can just as well set the reference to null before
you create the new buffer:

inputBuffer = null;
inputBuffer = new Byte[fs.Length];


I wouldn't allocate a new buffer for every file, though. I would use a
reasonably sized buffer to read chunks from the files:

while ((len = fs.Read(inputBu ffer, 0, 4096)) 0) {
fsOut.Write(inp utBuffer, 0, len);
}
fs.Close();

>A better solution for fs would be the using statment which would force GC
>and return the memory.

>
It wouldn't return the memory - but it *would* close/dispose the stream
in all situations, whether or not there's an exception.
Closing/disposing the stream doesn't return any memory, but it releases
the handle on the file.
>

[Göran Andersson]

Re: How does &quot;new&qu ot; work in a loop?

Jon Skeet [C# MVP] wrote:

Matt <matttelles@spr ynet.comwrote:
>
<snip>
>

>Unlikely that you are creating a memory leak. C# uses garbage
>collection. When the object goes out of scope (in your case, the }
>marked // for int i) the object is destroyed. The next time through
>the loop, a new one is created.

>
The object is *not* destroyed when it reaches the end of the scope.
.NET does not have deterministic garbage collection. Instead, the
object's memory will be released *at some point* after it is last used.
>
In fact, this could be before the end of the scope - the GC could kick
in before progBar.Value++ and free both fs and inputBuffer.
>

But at that point there are still references to those objects.

Actually, the buffer won't be collectable until after the next buffer
has been created in the next iteration of the loop, when the reference
is replaced by the reference to the new buffer.

[Barry Kelly]

Re: How does &quot;new&qu ot; work in a loop?

Göran Andersson <guffa@guffa.co mwrote:

In fact, this could be before the end of the scope - the GC could kick
in before progBar.Value++ and free both fs and inputBuffer.

>
But at that point there are still references to those objects.

Not if the variables are in registers and have been overwritten. The JIT
compiler can detect the variable's lifetime, it doesn't necessarily last
out the whole lexical scope.

Actually, the buffer won't be collectable until after the next buffer
has been created in the next iteration of the loop, when the reference
is replaced by the reference to the new buffer.

Ditto.

-- Barry

--

http://barrkel.blogspot.com/

[Göran Andersson]

Re: How does &quot;new&qu ot; work in a loop?

Barry Kelly wrote:

Göran Andersson <guffa@guffa.co mwrote:
>

>>In fact, this could be before the end of the scope - the GC could kick
>>in before progBar.Value++ and free both fs and inputBuffer.

>But at that point there are still references to those objects.

>
Not if the variables are in registers and have been overwritten. The JIT
compiler can detect the variable's lifetime, it doesn't necessarily last
out the whole lexical scope.

So you mean that the scope of a variable only last one single iteration
of the loop?

>Actually, the buffer won't be collectable until after the next buffer
>has been created in the next iteration of the loop, when the reference
>is replaced by the reference to the new buffer.

>
Ditto.
>
-- Barry
>

[Jon Skeet [C# MVP]]

Re: How does &quot;new&qu ot; work in a loop?

Göran Andersson wrote:

The object is *not* destroyed when it reaches the end of the scope.
.NET does not have deterministic garbage collection. Instead, the
object's memory will be released *at some point* after it is last used.

In fact, this could be before the end of the scope - the GC could kick
in before progBar.Value++ and free both fs and inputBuffer.

>
But at that point there are still references to those objects.
>
Actually, the buffer won't be collectable until after the next buffer
has been created in the next iteration of the loop, when the reference
is replaced by the reference to the new buffer.

Nope. In release mode, the JIT is smart enough to work out when a
variable can no longer be read, and will not count that variable as a
live root.

Jon

[Barry Kelly]

Re: How does &quot;new&qu ot; work in a loop?

Göran Andersson <guffa@guffa.co mwrote:

Barry Kelly wrote:

Göran Andersson <guffa@guffa.co mwrote:

>In fact, this could be before the end of the scope - the GC could kick
>in before progBar.Value++ and free both fs and inputBuffer.
But at that point there are still references to those objects.

Not if the variables are in registers and have been overwritten. The JIT
compiler can detect the variable's lifetime, it doesn't necessarily last
out the whole lexical scope.

>
So you mean that the scope of a variable only last one single iteration
of the loop?

I mean that the "FileStream fs" in the OP may be GCd before
progBar.Value++ , like Jon says. The variable's lifetime may be smaller
than its scope. Scope is a lexical concept that exists only at compile
time.

-- Barry

--

http://barrkel.blogspot.com/