how to read large CSV file

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • jx2
    New Member
    • Feb 2007
    • 228

    #1

    how to read large CSV file

    hi everyone
    i'm trying to read data from CSV file file is about 1MB (or bigger)
    i figured out how to read it quickly from the disc but it takes 3000 milliseconds to change it into arrays so my question is
    is there a faster way of doing it?
    thats my class:
    Code:
    public class ReadWrite {
        
        private static String filename = "myFile.csv";
        private static File file;
        private static Scanner in;
        private static String str="";
        private int[] time =     new int[100000];
        private float[] value =  new float[100000];
        private String [] date = new String[100000];
        private String[] update =new String[100000];
        
        public static int count = 0;
        public boolean hasNext = true;
        
        static private String str2="";
        
        /** Creates a new instance of ReadWrite */
        public ReadWrite() throws FileNotFoundException, IOException {
            file = new File(filename);
            fileReader = new FileReader(file);
            
            readFile();
            
            in = new Scanner(this.str);
            in.useDelimiter("(\t|\n)");
            
        }
    [B]// thats where the problems are this method is to slow
    // slower then PHP :/  
    //(in php i would use explode("\n",str) and explode("\t",str))[/B] 
        public void read(){
            while(in.hasNextInt()){
                time[count] = in.nextInt() ;
                value[count] = in.nextFloat();
                date[count] = in.next();
                update[count] = in.next();
                count++;
            }
        }
     [B]------------------------------------end of problems -----------------------------[/B]
        public String toString(int i){
            return time[i] + " " +value[i] +" "+ date[i];
        }
        
        FileReader fileReader;
        void readFile() throws IOException{
            while(hasNext){
                char [] c = new char[4096];
                if(fileReader.read(c) > -1 ){
                    str += String.valueOf(c);
                }else{
                    hasNext = false;                
                }
            }
        }
        
        
    }
  • BigDaddyLH
    Recognized Expert Top Contributor
    • Dec 2007
    • 1216

    #2
    I don't know about the speed of code, but first copying the entire file into a string is a whacky thing to do. Why not just read from the file?
    [CODE=Java]in = new Scanner(new File(filename)) ;[/CODE]
    Also, your use of static fields needs to be rethought, but on another day.

    Finally, why are you loading the entire file into memory at once? Can you avoid this?

    Comment

    • jx2
      New Member
      • Feb 2007
      • 228

      #3
      Originally posted by BigDaddyLH
      I don't know about the speed of code, but first copying the entire file into a string is a whacky thing to do. Why not just read from the file?
      [CODE=Java]in = new Scanner(new File(filename)) ;[/CODE]
      Also, your use of static fields needs to be rethought, but on another day.

      Finally, why are you loading the entire file into memory at once? Can you avoid this?
      actualy i used scanner but it was the slowest one
      perhaps i 'm missing something importent
      Code:
       str = in.nextLine();
      is there a better way to read whole file in ( i mean faster way)

      Comment

      • BigDaddyLH
        Recognized Expert Top Contributor
        • Dec 2007
        • 1216

        #4
        There are libraries for working with CSV files. Google for them.

        Comment

        • mattwarren
          New Member
          • Feb 2008
          • 1

          #5
          rather than use str += String.valueOf( c) try using a StringBuffer and use the append.

          something like:
          sb.append(Strin g.valueOf(c))

          Comment

          Working...