OutOfMemoryException When Loading Huge file into MemoryStream

Collapse
This topic is closed.
X
X
 
  • Time
  • Show
Clear All
new posts
  • Naamat

    #1

    OutOfMemoryException When Loading Huge file into MemoryStream

    Hello,

    I am the sample FPSEPublish
    (http://blog.baeke.info/blog/_archive.../3/393158.html) code to upload a
    document to Sharepoint (WSS). This works perfectly for samll documents.

    Problem:

    When I attempt to upload a huge document (300Megabayte) on a PC with 3.6Gig
    RAM I am getting a OutOfMemoryExce ption when the program attempts to read the
    file into a MemoryStream.

    The problem has nothing to do with Sharepoint as it is happening prior to
    the FP RPC upload atempt.

    Questions: Is there a limit on the amount of memory that is assigned to the
    GarbageCollecte d heap that is being reached? Using the Sysinternals
    ProcessExplorer program I can see the amount of memory being consumed by the
    process as follows:

    Total Reserved Bytes: 897,544,192

    I have tried to understand how the CLR is allocating memory to no avail.

    Any help/pointer is greatly appreciated

    Sample Code:

    +++++++++++++++ +++++++++++++++ +++++
    public void PutDocument(str ing uri, string fileName, string metaInfo)
    {
    //GC.Collect();
    //GC.WaitForPendi ngFinalizers();
    //GC.Collect();


    Uri myUri = new Uri(uri);
    string webUrl, fileUrl;
    UrlToWebUrl(uri , out webUrl, out fileUrl);

    if (null == metaInfo)
    metaInfo = "";

    if (!File.Exists(f ileName))
    throw new Exception("Coul d not find file" + fileName);

    string postBody = String.Format(
    "method=put+doc ument&service_n ame=&document=[document_name={ 0};meta_info=[{1}]]&put_option=ove rwrite&comment= &keep_checked_o ut=false\n",
    HttpUtility.Url Encode(fileUrl) ,
    metaInfo);

    ASCIIEncoding encoding = new ASCIIEncoding() ;
    MemoryStream stream = new MemoryStream();
    stream.Write(en coding.GetBytes (postBody), 0, postBody.Length );

    FileStream fs = File.OpenRead(f ileName);
    byte[] b = new byte[4096];
    while (fs.Read(b, 0, b.Length) > 0)
    {
    //GC.Collect();
    //GC.WaitForPendi ngFinalizers();
    //GC.Collect();

    try
    {
    stream.Write(b, 0, b.Length); //FAILS TO WRITE TO STREAM AFTER STREAM
    SIZE REACHES APPROX 270Meg
    //System.Diagnost ics.Debug.Write Line(stream.Len gth.ToString()) ;
    }
    catch(Exception ex)
    {
    System.Diagnost ics.Trace.Write Line(ex.Message );
    }
    }
    fs.Close();

    SendRequest(myU ri.GetLeftPart( UriPartial.Auth ority) + webUrl +
    "/_vti_bin/_vti_aut/author.dll", stream.GetBuffe r(), stream.Length);
    stream.Close();
    }


    +++++++++++++++ +++++++++++++++ +++++
  • Willy Denoyette [MVP]

    #2
    Re: OutOfMemoryExce ption When Loading Huge file into MemoryStream



    "Naamat" <simeon@newsgro up.nospam> wrote in message
    news:EB7E8740-7942-49E0-A09C-DCDE17E22B3C@mi crosoft.com...
    | Hello,
    |
    | I am the sample FPSEPublish
    | (http://blog.baeke.info/blog/_archive.../3/393158.html) code to
    upload a
    | document to Sharepoint (WSS). This works perfectly for samll documents.
    |
    | Problem:
    |
    | When I attempt to upload a huge document (300Megabayte) on a PC with
    3.6Gig
    | RAM I am getting a OutOfMemoryExce ption when the program attempts to read
    the
    | file into a MemoryStream.
    |
    | The problem has nothing to do with Sharepoint as it is happening prior to
    | the FP RPC upload atempt.
    |
    | Questions: Is there a limit on the amount of memory that is assigned to
    the
    | GarbageCollecte d heap that is being reached? Using the Sysinternals
    | ProcessExplorer program I can see the amount of memory being consumed by
    the
    | process as follows:
    |
    | Total Reserved Bytes: 897,544,192
    |
    | I have tried to understand how the CLR is allocating memory to no avail.
    |
    | Any help/pointer is greatly appreciated
    |
    | Sample Code:
    |
    | +++++++++++++++ +++++++++++++++ +++++
    | public void PutDocument(str ing uri, string fileName, string metaInfo)
    | {
    | //GC.Collect();
    | //GC.WaitForPendi ngFinalizers();
    | //GC.Collect();
    |
    |
    | Uri myUri = new Uri(uri);
    | string webUrl, fileUrl;
    | UrlToWebUrl(uri , out webUrl, out fileUrl);
    |
    | if (null == metaInfo)
    | metaInfo = "";
    |
    | if (!File.Exists(f ileName))
    | throw new Exception("Coul d not find file" + fileName);
    |
    | string postBody = String.Format(
    |
    "method=put+doc ument&service_n ame=&document=[document_name={ 0};meta_info=[{1}]]&put_option=ove rwrite&comment= &keep_checked_o ut=false\n",
    | HttpUtility.Url Encode(fileUrl) ,
    | metaInfo);
    |
    | ASCIIEncoding encoding = new ASCIIEncoding() ;
    | MemoryStream stream = new MemoryStream();
    | stream.Write(en coding.GetBytes (postBody), 0, postBody.Length );
    |
    | FileStream fs = File.OpenRead(f ileName);
    | byte[] b = new byte[4096];
    | while (fs.Read(b, 0, b.Length) > 0)
    | {
    | //GC.Collect();
    | //GC.WaitForPendi ngFinalizers();
    | //GC.Collect();
    |
    | try
    | {
    | stream.Write(b, 0, b.Length); //FAILS TO WRITE TO STREAM AFTER STREAM
    | SIZE REACHES APPROX 270Meg
    | //System.Diagnost ics.Debug.Write Line(stream.Len gth.ToString()) ;
    | }
    | catch(Exception ex)
    | {
    | System.Diagnost ics.Trace.Write Line(ex.Message );
    | }
    | }
    | fs.Close();
    |
    | SendRequest(myU ri.GetLeftPart( UriPartial.Auth ority) + webUrl +
    | "/_vti_bin/_vti_aut/author.dll", stream.GetBuffe r(), stream.Length);
    | stream.Close();
    | }
    |
    |
    | +++++++++++++++ +++++++++++++++ +++++

    |
    | Total Reserved Bytes: 897,544,192

    This doesn't mean a lot, but is this the reserved bytes before you attempted
    the load?
    Anyway, the exception means that you don't have a free contigious area of
    270MB available in your process 2GB Virtual address space.
    Also keep in mind that strings in .NET are really System.Char arrays, where
    a char is 16 bit. So if your doc is ASCII encoded, it will take twice the
    size of the file in memory.

    Willy.



    Comment

    • Jon Skeet [C# MVP]

      #3
      Re: OutOfMemoryExce ption When Loading Huge file into MemoryStream

      Willy Denoyette [MVP] <willy.denoyett e@telenet.be> wrote:

      <snip>
      [color=blue]
      > Also keep in mind that strings in .NET are really System.Char arrays, where
      > a char is 16 bit. So if your doc is ASCII encoded, it will take twice the
      > size of the file in memory.[/color]

      Where is the file being treated as a string? The ASCII encoding is only
      used for the URL, as far as I can see.

      --
      Jon Skeet - <skeet@pobox.co m>
      http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
      If replying to the group, please do not mail me too

      Comment

      • Jon Skeet [C# MVP]

        #4
        Re: OutOfMemoryExce ption When Loading Huge file into MemoryStream

        Naamat <simeon@newsgro up.nospam> wrote:[color=blue]
        > I am the sample FPSEPublish
        > (http://blog.baeke.info/blog/_archive.../3/393158.html) code to upload a
        > document to Sharepoint (WSS). This works perfectly for samll documents.[/color]

        No it doesn't, unfortunately.

        Aside from the memory problems you're getting, the code is actually
        wrong, and stylistically flawed too (I'm looking at the same code you
        pointed at).

        1) The code is always trying to read 4K at a time, and then *always*
        writing 4K whether it's read 1 byte of 4K. I suggest you have a look at
        Pobox has been discontinued as a separate service, and all existing customers moved to the Fastmail platform.


        2) It should also have a "using" statement around the FileStream (and
        preferrably the MemoryStream too, for consistency) to make sure it
        always gets disposed of when it's finished with, whether or not there
        was an exception. (This is a fault in other methods too.)

        3) Not an error, so much as bad style: It's using the
        if (constant==vari able) style which is a hangover from the bad-old days
        of C, where if you wrote if (variable=const ant) you wouldn't get a
        compiler error. The idiom leads to more reliable coding in C, but at
        the expense of a bit of readability.

        4) In SendRequest(str ing uri, byte[] postBody, long postLength),
        there's a lot of pointless looping around, writing 4K at a time, with
        no indication as to why he doesn't just write the whole lot in one go.


        Now, as to why you're running out of memory... I suspect the problem is
        with how the MemoryStream is growing. You could try creating the
        MemoryStream with enough space for the initial piece of text and
        everything you need from the post body's data (using Stream.Length).
        That would avoid MemoryStream having to copy everything each time it
        resizes.

        However, a better solution would be to refactor the code to avoid
        building up the MemoryStream in the first place. There's no reason why
        there shouldn't be a version of SendRequest which takes a stream to
        write stream to the request object. Now, whether that actually streams
        it to the client or whether it buffers it all up in memory anyway is
        another matter, but it would be saving at least one 300M+ buffer!

        --
        Jon Skeet - <skeet@pobox.co m>
        http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
        If replying to the group, please do not mail me too

        Comment

        • Willy Denoyette [MVP]

          #5
          Re: OutOfMemoryExce ption When Loading Huge file into MemoryStream


          "Jon Skeet [C# MVP]" <skeet@pobox.co m> wrote in message
          news:MPG.1e2cfb 053d74812b98cca 1@msnews.micros oft.com...
          | Willy Denoyette [MVP] <willy.denoyett e@telenet.be> wrote:
          |
          | <snip>
          |
          | > Also keep in mind that strings in .NET are really System.Char arrays,
          where
          | > a char is 16 bit. So if your doc is ASCII encoded, it will take twice
          the
          | > size of the file in memory.
          |
          | Where is the file being treated as a string? The ASCII encoding is only
          | used for the URL, as far as I can see.
          |
          |
          Not in this piece of code, but I guess that's not all there is running. It
          was just a warning, some may forget that when they threat the stream a
          strings they take twice the size.

          Willy.



          Comment

          • TerryFei

            #6
            RE: OutOfMemoryExce ption When Loading Huge file into MemoryStream

            Hi Simeon,

            I just wanted to check how things are going. If there is any question,
            please feel free to join the community and we are here to support you at
            your convenience. Thanks for your understanding!

            Best Regards,

            Terry Fei[MSFT]
            Microsoft Community Support
            Get Secure! www.microsoft.com/security
            (This posting is provided "AS IS", with no warranties, and confers no
            rights.)


            Comment

            Working...