Remove line breaks from a BIG text file

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • Chris9
    New Member
    • Mar 2010
    • 3

    Remove line breaks from a BIG text file

    I need to remove line breaks from a large ASCII table, but get a MemoryError. The data is an ASCII grid: the header takes up the first 6 lines followed by 22000 rows of data. I want to remove the line breaks from the table part of the file while leaving the first 6 line breaks. The following code works for small files but runs out of memory (at the 3rd line) for large files: (thanks to Zhaphod and Bvdet from an earlier post)
    Code:
    f1 = open(fn1) 
    f2 = open(fn2, 'w') 
    outputlist = [line.strip() for line in f1] 
    
    f2.write("\n".join(outputlist[:6]))
    f2.close() 
    f2 = open(fn2, 'a')
    
    f2.write("\n")
    f2.write(" ".join(outputlist[6:]))
    f1.close()
    f2.close()
    I thought that if i could do the .strip() for chunks of f1 then i could append each to the external file, but couldn't work out how.
  • bvdet
    Recognized Expert Specialist
    • Oct 2006
    • 2851

    #2
    Instead of reading the entire file into memory, iterate on the file object and incrementally write to the output file.
    Code:
    f1 = open("test1.txt")
    f2 = open("test1_output.txt", "w")
    for i, line in enumerate(f1):
        if i < 6:
            f2.write(line)
        else:
            f2.write(line.strip())
    f1.close()
    f2.close()

    Comment

    • Chris9
      New Member
      • Mar 2010
      • 3

      #3
      Thanks very much Bvdet. I had to add a line after line 7:
      f2.write(" ")

      Comment

      Working...