How to Read very large file with formatting (file size > 1 GB)

[Nelson Joseph]

Dear All,

Can anybody help me to read a very large file? and I need to maintain the formatting. The file size starts from 1 GB.

Is there any built-in library?

Thanks in advance

[weaknessforcats]

As long as you read the file onto the heap there should be no problem unless your system restricts maximum heap size per process. Just read the file.

[donbock]

You may need to open it as a binary file. It depends on the file format.

If you can't allocate a single heap buffer large enough for the entire file then you will be forced to look through the file a block at a time, trading reduced memory usage for more complicated software and increased execution time.

[Nelson Joseph]

Thank you for your guidance.
One more question...
my 1GB file has 5 columns. first column contains integer and second onwards float or double or integer.

e.g
32, 5.1275958839365 8e-030, 45.0, 0, 50.0

Please tell an efficient way to access each data. Could you share any code?

Thanks in advance

[donbock]

A 1GB file formatted as you describe is approximately 200,000 lines. Would you rather end up with a 1GB buffer holding the text of the file or four 200,000-element arrays holding the numeric values pulled from the file?

If you want the numeric values then you only need to repeatedly read a single line, parse it into the four numbers, and add that set of numbers to the arrays.

You should use a small test file with only a few lines to test your parsing software.