FileStream is too slow on huge files

[BlueVoodoo]

Hi,

I have a piece of code that needs to be able to modify a few bytes towards the end of a file. The problem is that the files are huge. Up to 100+ Gb.

I need the operation to be as fast as possible but after hours of Googeling, it looks like .Net is rather limited here???

I have mostly been trying using System.IO.FileS tream and know of no other methods. A "reverse" filestream would do but I have know idea how to create one (write from the end instead of the beginning).

Here is sort of what I do:

        static void Main(string[] args)
        {
            //Just to simulate a large file
            int size = 1000 * 1024 * 1024;
            string filename = "blah.dat";
            FileStream fs = new FileStream(filename, FileMode.Create);
            fs.SetLength(size);
            fs.Close();

            //Modify the last byte
            byte c = 255;
            fs = new FileStream(filename, FileMode.Open);
            
            //If I comment this out, the modification happens instantly
            fs.Seek(fs.Length - 1, SeekOrigin.Begin);
            fs.WriteByte(c);

            //Now, since I am modifying the last byte, this last step is extremely slow on large files (as slow as copying the entire file)
            fs.Close();
        }

[Aimee Bailey]

I believe that this is happening because your asking the FileStream to seek to the end of the file, however if you seek using an offset of -1, it should seek backwards from the end of the file.

hope this helps.

Aimee.

[BlueVoodoo]

I believe that this is happening because your asking the FileStream to seek to the end of the file, however if you seek using an offset of -1, it should seek backwards from the end of the file.

hope this helps.

Aimee.

Tried it but I get an IOException trying to seek with a negative value.

"An attempt was made to move the file pointer before the beginning of the file."

[Aimee Bailey]

I just tested it, works fine :), heres how i achieved it:

private void button1_Click(object sender, EventArgs e)
{
  OpenFileDialog dlg = new OpenFileDialog();
  if (dlg.ShowDialog() == DialogResult.OK)
  {
    using (FileStream f = new FileStream(dlg.FileName,
                                         FileMode.Open,
                                         FileAccess.Read))
    {
       f.Seek(-1, SeekOrigin.End);
       f.ReadByte();
       string result = String.Format("Position: {0} \r\n" +
                                     "Length Of File: {1}",
                                     fs.Position.ToString(),
                                     fs.Length.ToString());
       MessageBox.Show(result);
    }
  }
}

[BlueVoodoo]

I just tested it, works fine :), heres how i achieved it:

private void button1_Click(object sender, EventArgs e)
{
  OpenFileDialog dlg = new OpenFileDialog();
  if (dlg.ShowDialog() == DialogResult.OK)
  {
    using (FileStream f = new FileStream(dlg.FileName,
                                         FileMode.Open,
                                         FileAccess.Read))
    {
       f.Seek(-1, SeekOrigin.End);
       f.ReadByte();
       string result = String.Format("Position: {0} \r\n" +
                                     "Length Of File: {1}",
                                     fs.Position.ToString(),
                                     fs.Length.ToString());
       MessageBox.Show(result);
    }
  }
}

Right, so seeking itself is not the issue. Reading the end of a huge file is fast. It's writing to the end of the file that is super slow. The time is spent when you close the filestream.

fs.Seek(-1, SeekOrigin.End) ;
and
fs.Seek(fs.Leng th - 1, SeekOrigin.Begi n);
is both very fast.

[BlueVoodoo]

Seems we have an answer.

http://stackoverflow.com/questions/3222305/system-io-filestream-is-super-slow-on-huge-files

See the reply from Darin Dimitrov.