ReadXML OutOfMemoryException

Collapse
This topic is closed.
X
X
Collapse
 
  • Page of 1
  • Time
  • Show
Clear All
new posts
Previous template Next
  • joey.powell@topscene.com
    #1

    ReadXML OutOfMemoryException

    Hello guys, I have a pretty simple app that reads XML data (.ReadXML)
    into an in-memory dataset, runs some linq queries, mods the data, and
    then writes the XML (.WriteXML) back to disk.

    Everything works well until, occasionally, I get an
    OutOfMemoryExce ption. So far this has only happened for a file that
    was about 50MB in size. All other, smaller files are working just
    fine.

    50MB (+overhead of other objects etc...) certainly does not seem like
    an unreasonable amount of data to hold in memory. How can I get this
    to work without the error? Can I manually allocate more memory for the
    DataSet and .ReadXML? Why doesn't it take advantage of more of the
    available memory?

    I know that there are lots of other ways to do this...but I am not
    interested in reengineering the code. I just want it to work, because
    I have since moved on to other projects. Is there a "quick and dirty"
    way to make my app able to handle the 50MB of data in memory? This
    does not seem like it should be such a big deal.

    Apparently the .ReadXML and .WriteXML methods of the DataSet are not
    very scalable!?
    Tags: None
    • cfps.Christian
      #2
      Re: ReadXML OutOfMemoryExce ption

      I'm actually fighting a bug that has something to do with this. I was
      told that a few years ago Microsoft "fixed" something that made it so
      their file transfer buffer is somewhere around 50MB. What this means
      to a developer is there is a chance for a System exception if you try
      to move more than that.

      I haven't tested my code yet but right now I've got it altered so it
      only moves 50MB at a time to hopefully get around this problem.

        Comment

        • Moe Sisko
          #3
          Re: ReadXML OutOfMemoryExce ption

          In the past, I've resorted to reading the big file a line at a time, and
          breaking it down into smaller sized temp files. A low tech solution, and
          you may need to be careful when splitting up the file depending on what
          assumptions you can make about the format of the big file.

          Moe

          <joey.powell@to pscene.comwrote in message
          news:bdf22521-f798-44c9-b151-549b3a1d3a46@x3 5g2000hsb.googl egroups.com...
          Hello guys, I have a pretty simple app that reads XML data (.ReadXML)
          into an in-memory dataset, runs some linq queries, mods the data, and
          then writes the XML (.WriteXML) back to disk.
          >
          Everything works well until, occasionally, I get an
          OutOfMemoryExce ption. So far this has only happened for a file that
          was about 50MB in size. All other, smaller files are working just
          fine.
          >

            Comment

            Previous template Next
            Working...