Regex - Memory performance

[Paul Hatcher]

Re: Regex - Memory performance

Willy

One other possibility is that the RegEx objects are reaching Gen2 before
being dereferenced, so they won't be picked up until it does a Gen2 collect.
It won't force a Gen2 collect until it hits the memory high watermark (32mb
free ISTR) as it's cheaper to just keep on allocating memory until something
else needs it.

It's confusing behaviour the first time you see it; process memory in
TaskManager just keeps getting higher and higher for no apparent reason.

Regards

Paul

"Willy Denoyette [MVP]" <willy.denoyett e@telenet.be> wrote in message
news:ug%23ItZzw FHA.2620@TK2MSF TNGP09.phx.gbl. ..[color=blue]
>
> <jeevankodali@g mail.com> wrote in message
> news:1127787056 .983282.157300@ f14g2000cwb.goo glegroups.com.. .[color=green]
>> my two lines of code is similar to the actual situation of the program
>> I am running. And it is similar to what you have written (as an example
>> code). The .Net Memory profiler which I am using shows when GC0, GC1
>> and GC2 kicks off and shows memory going down etc. But the memory used
>> by Regex objects is not going down. Thats what I was trying to say. Its
>> like I am runnign the example code written by you and Regex objects are
>> not being collected.
>>[/color]
>
> Did you actually run the code I showed? Did you check the GC counters when
> it runs?
> If the RegEx objects aren't collected you will see memory usage going up
> without ever going down, resulting in an OM exception, but it's not how
> the program behaves. Again forget about the memory profiler, run the
> program and watch the performance counters.
>
>
> Willy.
>
>[/color]

[Ludovic SOEUR]

Re: Regex - Memory performance

To be sure there were no memory leak, I wrote 2 simple programs. The first
one keep a reference on the regex to be sure there is a memory leak. The
second one is only a loop that should not have a memory leak.

I profiled the results with .Net Memory Profiler. For each program, I took 2
snapshots when you must press a key (on the line System.Console. Read()) and
asked memory profiler to show differences. The results were logical : in the
first case, there is a leak, and in the second case, there is no leak.


------------------------------------------------
First code that have a memory leak :

using System;
using System.Collecti ons;
using System.Text.Reg ularExpressions ;

class Class1 {
static void Main() {
Hashtable h=new Hashtable();
for(int j=0;j<3;j++) {
System.Console. WriteLine("Star t");
for(int i=0;i<1000;i++) {
Regex regex=new Regex("^myPatte rn "+i+"$",RegexOp tions.IgnoreCas e);
Match match=regex.Mat ch("myInput "+i);
Console.WriteLi ne("Regex = "+regex.ToStrin g()+"\tMatch =
"+match.Success );
h.Add(j*10000+i ,regex);
}
System.Console. WriteLine("End" );
System.Console. Read();
}
}
}

Results :
System.Text.Reg ularExpression. Regex : Delta = 2000
System.String : Delta = 2000
System.Int32 : Delta = 2000


---------------------------------------------------------------
Second code that have no memory leak :

using System;
using System.Text.Reg ularExpressions ;

class Class1 {
static void Main() {
for(int j=0;j<3;j++) {
System.Console. WriteLine("Star t");
for(int i=0;i<1000;i++) {
Regex regex=new Regex("^myPatte rn "+i+"$",RegexOp tions.IgnoreCas e);
Match match=regex.Mat ch("myInput "+i);
Console.WriteLi ne("Regex = "+regex.ToStrin g()+"\tMatch =
"+match.Success );
}
System.Console. WriteLine("End" );
System.Console. Read();
}
}
}

Results :
System.Text.Reg ularExpression. Regex : Delta = 0
System.String : Delta = 0
System.Int32 : Delta = 0

-----------------------------------------------------------

The loop is as simple as the one you told you had :
for(int i = 0; i < length; i++) {
Regex r = new Regex(pattern, RegexOptions.Ig noreCase);
}

My conclusion is that there is no memory leak with Regex with your loop. You
may have a reference somewhere.
Again, put you ENTIRE source code. It's impossible to help you without ALL
your source. Maybe you think there is no reference to your regex but we can
find out if there is one. How do you initialise your regex, how do you do
your match, .....

Try my two examples to see if there is a leak on your computer. If there is
a leak for the second example it is maybe because you don't use properly
memory profiler....

Hope it helps,

Ludovic SOEUR.


<jeevankodali@g mail.com> a écrit dans le message de
news:1127787056 .983282.157300@ f14g2000cwb.goo glegroups.com.. .[color=blue]
> my two lines of code is similar to the actual situation of the program
> I am running. And it is similar to what you have written (as an example
> code). The .Net Memory profiler which I am using shows when GC0, GC1
> and GC2 kicks off and shows memory going down etc. But the memory used
> by Regex objects is not going down. Thats what I was trying to say. Its
> like I am runnign the example code written by you and Regex objects are
> not being collected.
>[/color]

[Willy Denoyette [MVP]]

Re: Regex - Memory performance


"Paul Hatcher" <phatcher@nospa m.cix.co.uk> wrote in message
news:edsy2c1wFH A.2620@TK2MSFTN GP09.phx.gbl...[color=blue]
> Willy
>
> One other possibility is that the RegEx objects are reaching Gen2 before
> being dereferenced, so they won't be picked up until it does a Gen2
> collect. It won't force a Gen2 collect until it hits the memory high
> watermark (32mb free ISTR) as it's cheaper to just keep on allocating
> memory until something else needs it.
>
> It's confusing behaviour the first time you see it; process memory in
> TaskManager just keeps getting higher and higher for no apparent reason.
>
> Regards
>
> Paul
>[/color]

Paul,

When you run the sample I posted, you'll notice a high collection rate for
all generations, the allocation rate on v1.1 is much higher than the gen0
collection frequency, that means there are a lot of promotions. The same
goes for gen1, so a lot objects are reaching gen2 (the number depends on CPU
performance, the type of GC, OS version etc.). However, one of the GC
heuristics forces a gen2 collection every x gen1 GC runs, where x varies
depending on the allocation rate measured in the previous cycle. What I
notice when I run the sample, is a gen2 collection per ~10 gen1 collections
( ~1 gen2 collection per second), the gen2 heap never exceeding 2Mb and the
working set never exeeds 12MB. You can try yourself, you'll see the memory
never reaches the high water mark you mentioned.

Willy.

[jeevankodali@gmail.com]

Re: Regex - Memory performance

Since I started this topic, I think its my responsibility to say that I
am getting out of this topic. Reason is that I cannot post my code (and
I am more than 100% sure that my example accurately describes what I
have with my source code). I fixed my problem (with the hashtable
method). If no one thinks there is no problem with Regex, then its
fine. But I learned one valuable lesson on where to look for memory
growth, if there is one.

Thanks for all your replies.

[Willy Denoyette [MVP]]

Re: Regex - Memory performance

After several request for a complete sample that illustrates the issue, you
are telling us that you can't post your code. I posted a complete sample
(essentially the same your code as you said), that illustrates there is no
such issue, at least not when I ran it, did you actually run this code?
guess not, did you look at the performance counters? guess not.
As long as you can't prove your issue with Regex, I will assume there is no
issue at all.

Willy.



<jeevankodali@g mail.com> wrote in message
news:1127860817 .116908.68370@g 14g2000cwa.goog legroups.com...[color=blue]
> Since I started this topic, I think its my responsibility to say that I
> am getting out of this topic. Reason is that I cannot post my code (and
> I am more than 100% sure that my example accurately describes what I
> have with my source code). I fixed my problem (with the hashtable
> method). If no one thinks there is no problem with Regex, then its
> fine. But I learned one valuable lesson on where to look for memory
> growth, if there is one.
>
> Thanks for all your replies.
>[/color]

[Ludovic SOEUR]

Re: Regex - Memory performance

In the previous posts, you were asking about Regex :[color=blue]
>I am really interested (if there is one reasoning) to know why this is
>happening. In the sense, why these objects are not being removed from
>memory. If this is a random thing which is out of our hands (and so in[/color]
t>he hands of people at Microsoft) then that answers everything.

I posted to you two samples that do what you said you were doing to
let you try and understand where there is a problem. You did not reply
that means you did not tried.

To me, the anser is : there is no memory leak with Regex and I will continue
to think that until someone shows me an example of leaking that I can
compile.
If you still think there is, I ,certainly Willy too and others of this
newsgroup,
ARE INTERESTED to have a source code that prove it.

I'm still interested in knowing the issue of this problem,
Ludovic.