a leaky program...

Collapse
This topic is closed.
X
X
 
  • Time
  • Show
Clear All
new posts
  • Casual Reader

    a leaky program...

    Hi,

    I have a C# program that in principle should only use a constant amount of
    memory to execute (with periodic garbage collection) and in fact does so when
    executed in Mono under either Windows or Linux. However, with Microsoft's
    own .NET execution environment, the program rapidly takes up more and more
    memory until it dies with an out-of-memory exception.

    I post an example code below, in the hope that someone with more knowledge
    of C#/.NET could enlighten me on some possible causes of such leaky behavior.

    Thanks.

    ------------------------------------------------------------------------------------------------

    using System;
    using System.Data.Sql Client;
    using System.Security .Cryptography;

    /*
    * Prerequisites:
    * - On (local), need to have DB named "test_db" with table named "gotera",
    * or adjust connection and query strings to match own settings.
    * - The table can have any schema but needs to have sufficient number of
    * rows (say, 1000000) for the program to run for more than a few seconds.
    *
    * No leaky behavior is observed if:
    * - The [STAThread] tag is removed;
    * - The MD5 construction is removed; or
    * - The enumeration of DB table rows is replaced with a simple loop
    * such as "for (int i=0; i<100000; ++i) { ... }".
    */

    class Leaky
    {
    [STAThread]
    static int Main() {
    long n_RowCnt = 0;
    SqlConnection conn = new
    SqlConnection(@ "server=(local) ;database=test_ db;trusted_conn ection=yes");
    try {
    conn.Open();
    SqlCommand cmd = new SqlCommand("sel ect * from gotera", conn);
    SqlDataReader reader = cmd.ExecuteRead er();
    while (reader.Read()) {
    ++n_RowCnt;
    MD5CryptoServic eProvider md5 = new MD5CryptoServic eProvider();
    }
    }
    catch (Exception ex) {
    System.Console. WriteLine(ex.Me ssage);
    }
    finally {
    conn.Close();
    }
    System.Console. WriteLine("Coun ted {0} rows.", n_RowCnt);
    return 0;
    }
    }

  • Jon Skeet [C# MVP]

    #2
    Re: a leaky program...

    Casual Reader <CasualReader@d iscussions.micr osoft.com> wrote:[color=blue]
    > I have a C# program that in principle should only use a constant amount of
    > memory to execute (with periodic garbage collection) and in fact does so when
    > executed in Mono under either Windows or Linux. However, with Microsoft's
    > own .NET execution environment, the program rapidly takes up more and more
    > memory until it dies with an out-of-memory exception.
    >
    > I post an example code below, in the hope that someone with more knowledge
    > of C#/.NET could enlighten me on some possible causes of such leaky behavior.[/color]

    One thing you're not doing is disposing of either the command (which
    shouldn't be causing a leak as there's only one of them) or the
    MD5CryptoServic eProvider. I'm surprised it's causing a leak, but it's
    something to try...

    --
    Jon Skeet - <skeet@pobox.co m>
    Pobox has been discontinued as a separate service, and all existing customers moved to the Fastmail platform.

    If replying to the group, please do not mail me too

    Comment

    • Willy Denoyette [MVP]

      #3
      Re: a leaky program...


      "Casual Reader" <CasualReader@d iscussions.micr osoft.com> wrote in message
      news:3AFA9C7C-FC15-4A8A-819D-00481E8B7BDC@mi crosoft.com...[color=blue]
      > Hi,
      >
      > I have a C# program that in principle should only use a constant amount of
      > memory to execute (with periodic garbage collection) and in fact does so
      > when
      > executed in Mono under either Windows or Linux. However, with Microsoft's
      > own .NET execution environment, the program rapidly takes up more and more
      > memory until it dies with an out-of-memory exception.
      >
      > I post an example code below, in the hope that someone with more knowledge
      > of C#/.NET could enlighten me on some possible causes of such leaky
      > behavior.
      >
      > Thanks.
      >
      > ------------------------------------------------------------------------------------------------
      >
      > using System;
      > using System.Data.Sql Client;
      > using System.Security .Cryptography;
      >
      > /*
      > * Prerequisites:
      > * - On (local), need to have DB named "test_db" with table named "gotera",
      > * or adjust connection and query strings to match own settings.
      > * - The table can have any schema but needs to have sufficient number of
      > * rows (say, 1000000) for the program to run for more than a few
      > seconds.
      > *
      > * No leaky behavior is observed if:
      > * - The [STAThread] tag is removed;
      > * - The MD5 construction is removed; or
      > * - The enumeration of DB table rows is replaced with a simple loop
      > * such as "for (int i=0; i<100000; ++i) { ... }".
      > */
      >
      > class Leaky
      > {
      > [STAThread]
      > static int Main() {
      > long n_RowCnt = 0;
      > SqlConnection conn = new
      > SqlConnection(@ "server=(local) ;database=test_ db;trusted_conn ection=yes");
      > try {
      > conn.Open();
      > SqlCommand cmd = new SqlCommand("sel ect * from gotera", conn);
      > SqlDataReader reader = cmd.ExecuteRead er();
      > while (reader.Read()) {
      > ++n_RowCnt;
      > MD5CryptoServic eProvider md5 = new MD5CryptoServic eProvider();
      > }
      > }
      > catch (Exception ex) {
      > System.Console. WriteLine(ex.Me ssage);
      > }
      > finally {
      > conn.Close();
      > }
      > System.Console. WriteLine("Coun ted {0} rows.", n_RowCnt);
      > return 0;
      > }
      > }
      >[/color]


      The reason relates to the apartment context your main thread runs in, Mono
      doesn't have any apartment concept, that's the reason why you don't se this
      happen.

      What's happening is the following, your MD5CryptoServic eProvider is never
      disposed of, so you are relying on the finalizer to clean up the unmanaged
      resources when he runs MD5CryptoServic eProvider Finalize method.
      Now, whenever you specify [STAThread] on "Main", your thread enters a STA ,
      however, the finalizer thread, who runs in a MTA, can't directly call
      Finalize without marshaling the call between both apartments/threads.
      In order to successfully marshal a call, an STA thread must pump messages
      f.i by calling DoEvents(), failing to do so will block the finalizer.

      There are two possible options to solve this:

      1. Call DoEvents(),
      while (reader.Read()) {
      DoEvents();
      ..

      or much better...

      2. Remove the STAThread attribute AND wrap your
      MD5CryptoServic eProvider in a using block like this/

      using(MD5Crypto ServiceProvider md5 = ...)
      {
      . .
      }

      Not sure why you need another instance of md5 for each iteration though.

      Willy.



      Comment

      • Casual Reader

        #4
        Re: a leaky program...

        Hi,

        Thanks to both Jon Skeet and Willy Denoyette for contributing their insights
        to my problem.

        As I understand the comments of both people, it is a bad interaction between
        "STA" and "MTA" that exists between my program and the execution environment;
        one remedy is to explicitly dispose of objects that are no longer required.

        In response to a comment re: why a new MD5 object was instantiated per
        iteration, I found very little documentation on the proper usage of MD5
        objects in .NET when I wrote the program. The one example that I found was
        Microsoft Knowledgebase Article 307020; bullet point 6 of that article says
        to use a new MD5 object for each new MD5 that you want to compute.

        Finally, one nit-picking comment that deserves no response: When I was
        investigating why my program was leaking memory, I narrowed down the problem
        to an interplay between three things in my program: the [STAThread] tag, the
        repeated instantiation of the MD5 object, and the enumeration of rows of a DB
        table. If I removed any one of the three, I would get no leaky behavior;
        that's why I posted the example code. But I've already learned a few things
        from the two MVPs, and I'm content for now...

        Thanks and regards.


        "Willy Denoyette [MVP]" wrote:
        [color=blue]
        >
        > "Casual Reader" <CasualReader@d iscussions.micr osoft.com> wrote in message
        > news:3AFA9C7C-FC15-4A8A-819D-00481E8B7BDC@mi crosoft.com...[color=green]
        > > Hi,
        > >
        > > I have a C# program that in principle should only use a constant amount of
        > > memory to execute (with periodic garbage collection) and in fact does so
        > > when
        > > executed in Mono under either Windows or Linux. However, with Microsoft's
        > > own .NET execution environment, the program rapidly takes up more and more
        > > memory until it dies with an out-of-memory exception.
        > >
        > > I post an example code below, in the hope that someone with more knowledge
        > > of C#/.NET could enlighten me on some possible causes of such leaky
        > > behavior.
        > >
        > > Thanks.
        > >
        > > ------------------------------------------------------------------------------------------------
        > >
        > > using System;
        > > using System.Data.Sql Client;
        > > using System.Security .Cryptography;
        > >
        > > /*
        > > * Prerequisites:
        > > * - On (local), need to have DB named "test_db" with table named "gotera",
        > > * or adjust connection and query strings to match own settings.
        > > * - The table can have any schema but needs to have sufficient number of
        > > * rows (say, 1000000) for the program to run for more than a few
        > > seconds.
        > > *
        > > * No leaky behavior is observed if:
        > > * - The [STAThread] tag is removed;
        > > * - The MD5 construction is removed; or
        > > * - The enumeration of DB table rows is replaced with a simple loop
        > > * such as "for (int i=0; i<100000; ++i) { ... }".
        > > */
        > >
        > > class Leaky
        > > {
        > > [STAThread]
        > > static int Main() {
        > > long n_RowCnt = 0;
        > > SqlConnection conn = new
        > > SqlConnection(@ "server=(local) ;database=test_ db;trusted_conn ection=yes");
        > > try {
        > > conn.Open();
        > > SqlCommand cmd = new SqlCommand("sel ect * from gotera", conn);
        > > SqlDataReader reader = cmd.ExecuteRead er();
        > > while (reader.Read()) {
        > > ++n_RowCnt;
        > > MD5CryptoServic eProvider md5 = new MD5CryptoServic eProvider();
        > > }
        > > }
        > > catch (Exception ex) {
        > > System.Console. WriteLine(ex.Me ssage);
        > > }
        > > finally {
        > > conn.Close();
        > > }
        > > System.Console. WriteLine("Coun ted {0} rows.", n_RowCnt);
        > > return 0;
        > > }
        > > }
        > >[/color]
        >
        >
        > The reason relates to the apartment context your main thread runs in, Mono
        > doesn't have any apartment concept, that's the reason why you don't se this
        > happen.
        >
        > What's happening is the following, your MD5CryptoServic eProvider is never
        > disposed of, so you are relying on the finalizer to clean up the unmanaged
        > resources when he runs MD5CryptoServic eProvider Finalize method.
        > Now, whenever you specify [STAThread] on "Main", your thread enters a STA ,
        > however, the finalizer thread, who runs in a MTA, can't directly call
        > Finalize without marshaling the call between both apartments/threads.
        > In order to successfully marshal a call, an STA thread must pump messages
        > f.i by calling DoEvents(), failing to do so will block the finalizer.
        >
        > There are two possible options to solve this:
        >
        > 1. Call DoEvents(),
        > while (reader.Read()) {
        > DoEvents();
        > ..
        >
        > or much better...
        >
        > 2. Remove the STAThread attribute AND wrap your
        > MD5CryptoServic eProvider in a using block like this/
        >
        > using(MD5Crypto ServiceProvider md5 = ...)
        > {
        > . .
        > }
        >
        > Not sure why you need another instance of md5 for each iteration though.
        >
        > Willy.
        >
        >
        >
        >[/color]

        Comment

        • Willy Denoyette [MVP]

          #5
          Re: a leaky program...

          Inline **

          Willy.

          "Casual Reader" <CasualReader@d iscussions.micr osoft.com> wrote in message
          news:54AD0BD4-EB4E-4116-86E8-59F8314670E1@mi crosoft.com...[color=blue]
          > Hi,
          >
          > Thanks to both Jon Skeet and Willy Denoyette for contributing their
          > insights
          > to my problem.
          >
          > As I understand the comments of both people, it is a bad interaction
          > between
          > "STA" and "MTA" that exists between my program and the execution
          > environment;
          > one remedy is to explicitly dispose of objects that are no longer
          > required.
          >[/color]

          ** The real source of the problem here is - a failure to pump messages on a
          STA thread - which makes it impossible for the Finalizer thread, that runs
          in a MTA, to marshal the calls to the Finalize method on the STA thread.
          Therefore, I suggest you only apply STAThread on a Windows (Forms) program
          "Main" entry. In all other cases, if you need a STA thread for COM interop,
          make sure you pump the message queue.
          [color=blue]
          > In response to a comment re: why a new MD5 object was instantiated per
          > iteration, I found very little documentation on the proper usage of MD5
          > objects in .NET when I wrote the program. The one example that I found
          > was
          > Microsoft Knowledgebase Article 307020; bullet point 6 of that article
          > says
          > to use a new MD5 object for each new MD5 that you want to compute.
          >[/color]
          ** That's right.
          [color=blue]
          > Finally, one nit-picking comment that deserves no response: When I was
          > investigating why my program was leaking memory, I narrowed down the
          > problem
          > to an interplay between three things in my program: the [STAThread] tag,
          > the
          > repeated instantiation of the MD5 object, and the enumeration of rows of a
          > DB
          > table. If I removed any one of the three, I would get no leaky behavior;
          > that's why I posted the example code. But I've already learned a few
          > things
          > from the two MVPs, and I'm content for now...
          >
          > Thanks and regards.
          >
          >
          > "Willy Denoyette [MVP]" wrote:
          >[color=green]
          >>
          >> "Casual Reader" <CasualReader@d iscussions.micr osoft.com> wrote in message
          >> news:3AFA9C7C-FC15-4A8A-819D-00481E8B7BDC@mi crosoft.com...[color=darkred]
          >> > Hi,
          >> >
          >> > I have a C# program that in principle should only use a constant amount
          >> > of
          >> > memory to execute (with periodic garbage collection) and in fact does
          >> > so
          >> > when
          >> > executed in Mono under either Windows or Linux. However, with
          >> > Microsoft's
          >> > own .NET execution environment, the program rapidly takes up more and
          >> > more
          >> > memory until it dies with an out-of-memory exception.
          >> >
          >> > I post an example code below, in the hope that someone with more
          >> > knowledge
          >> > of C#/.NET could enlighten me on some possible causes of such leaky
          >> > behavior.
          >> >
          >> > Thanks.
          >> >
          >> > ------------------------------------------------------------------------------------------------
          >> >
          >> > using System;
          >> > using System.Data.Sql Client;
          >> > using System.Security .Cryptography;
          >> >
          >> > /*
          >> > * Prerequisites:
          >> > * - On (local), need to have DB named "test_db" with table named
          >> > "gotera",
          >> > * or adjust connection and query strings to match own settings.
          >> > * - The table can have any schema but needs to have sufficient number
          >> > of
          >> > * rows (say, 1000000) for the program to run for more than a few
          >> > seconds.
          >> > *
          >> > * No leaky behavior is observed if:
          >> > * - The [STAThread] tag is removed;
          >> > * - The MD5 construction is removed; or
          >> > * - The enumeration of DB table rows is replaced with a simple loop
          >> > * such as "for (int i=0; i<100000; ++i) { ... }".
          >> > */
          >> >
          >> > class Leaky
          >> > {
          >> > [STAThread]
          >> > static int Main() {
          >> > long n_RowCnt = 0;
          >> > SqlConnection conn = new
          >> > SqlConnection(@ "server=(local) ;database=test_ db;trusted_conn ection=yes");
          >> > try {
          >> > conn.Open();
          >> > SqlCommand cmd = new SqlCommand("sel ect * from gotera", conn);
          >> > SqlDataReader reader = cmd.ExecuteRead er();
          >> > while (reader.Read()) {
          >> > ++n_RowCnt;
          >> > MD5CryptoServic eProvider md5 = new MD5CryptoServic eProvider();
          >> > }
          >> > }
          >> > catch (Exception ex) {
          >> > System.Console. WriteLine(ex.Me ssage);
          >> > }
          >> > finally {
          >> > conn.Close();
          >> > }
          >> > System.Console. WriteLine("Coun ted {0} rows.", n_RowCnt);
          >> > return 0;
          >> > }
          >> > }
          >> >[/color]
          >>
          >>
          >> The reason relates to the apartment context your main thread runs in,
          >> Mono
          >> doesn't have any apartment concept, that's the reason why you don't se
          >> this
          >> happen.
          >>
          >> What's happening is the following, your MD5CryptoServic eProvider is never
          >> disposed of, so you are relying on the finalizer to clean up the
          >> unmanaged
          >> resources when he runs MD5CryptoServic eProvider Finalize method.
          >> Now, whenever you specify [STAThread] on "Main", your thread enters a STA
          >> ,
          >> however, the finalizer thread, who runs in a MTA, can't directly call
          >> Finalize without marshaling the call between both apartments/threads.
          >> In order to successfully marshal a call, an STA thread must pump messages
          >> f.i by calling DoEvents(), failing to do so will block the finalizer.
          >>
          >> There are two possible options to solve this:
          >>
          >> 1. Call DoEvents(),
          >> while (reader.Read()) {
          >> DoEvents();
          >> ..
          >>
          >> or much better...
          >>
          >> 2. Remove the STAThread attribute AND wrap your
          >> MD5CryptoServic eProvider in a using block like this/
          >>
          >> using(MD5Crypto ServiceProvider md5 = ...)
          >> {
          >> . .
          >> }
          >>
          >> Not sure why you need another instance of md5 for each iteration though.
          >>
          >> Willy.
          >>
          >>
          >>
          >>[/color][/color]


          Comment

          Working...