remove duplicates from file

[topala1984]

hello all

excuse my english, I`m a beginer in java and I want to remove duplicate cities + rearrenging file`s lines elements from one 150 mb file, in my file I have that:

"68586240","685 86367","IRVING"
"68586368","685 86431","DENTON"
"68586432","685 86495","DENTON"

and I want to become

"68586240","685 86367","IRVING"
"68586368","685 86495","DENTON"

i mean

1 2 x
3 4 x
5 6 y

become

1 4 x
5 6 y

I have allready done the program but i`m stuck . can anyone help me?

import java.util.regex.*;
import java.io.*;

	public class test {

	    private static final String REGEX = ",";
	    private static Pattern pattern;
		
		 public static void main(String[] argv) {
		 	
		 	pattern = Pattern.compile(REGEX);
		 		
			try{
				FileInputStream fstream = new FileInputStream("test.txt");
				DataInputStream in = new DataInputStream(fstream);
				BufferedReader br = new BufferedReader(new InputStreamReader(in));
				String strLine;
							
				int i = 1;
				
				
				BufferedWriter make_file = new BufferedWriter(new FileWriter("file_temp.txt", true));
				String make_test = "\"TEST\",\"TEST\",\"TEST\"";
				make_file.write(make_test);
				make_file.close();
				
    			
    			BufferedWriter out = new BufferedWriter(new FileWriter("good.txt", true));    			
    			
				while ((strLine = br.readLine()) != null)   {
					
					//explode line					
    				String[] items = pattern.split(strLine);
    				
    				// read + memory content from temp file	
					FileReader input = new FileReader("file_temp.txt");
					BufferedReader bufRead = new BufferedReader(input);
			       	String line = ""; 
			        String temp = "";	
                    line = bufRead.readLine();
					while (line != null){
            		    temp = temp+line;
                		line = bufRead.readLine();                
            		}
                    bufRead.close();
					
					// explode line from temp file
					String[] items_temp = pattern.split(temp);
					
					//delete temp file
					new	File("file_temp.txt").delete();
						
    				BufferedWriter out_temp = new BufferedWriter(new FileWriter("file_temp.txt", true));
					out_temp.write(strLine);
					out_temp.close();
					
					String x1 = items_temp[2];
					String x2 = items[2];
					
					if (x1.equalsIgnoreCase(x2)) {
						// ??????????
					} 
					else {
						out.write(items[0]+","+items_temp[1]+","+items[2]+"\n");
						/* 
						 input   output  correct 
						 1 2 x   1 2 x   1 4 x
						 3 4 x   5 6 y   5 6 y
						 5 6 y
						*/
						
					}
					
					System.out.println(i+"-"+strLine);			
					i++;

				}
			 	out.close();
				in.close();
			}
			catch (Exception e){
				System.err.println("Error: " + e.getMessage());
			}	
			
	}
	
}

tnx

[dmjpro]

Sorry for less time in my hand to look after your code throughly :-)
Have a look at my code ....

[code=java]
BufferedReader in = new BufferedReader( new FileReader("src _file"));
FileWriter out = new FileWriter("tgt _file");
String line;
HashSet hs = new HashSet();
String op = "";
while(!(line=in .readLine())!=n ull){
StringTokenizer s = new StringTokenizer (line,",");
String city;
while(s.hasMore Tokes()) city = s.nextToken();
op += hs.add(city) ? (line+System.ge tProperty("line .separator")) : "";
//here it ignores duplicate items
}
out.write(op);
in.close();
op.close();
[/code]

Enjoy this code.
Good Luck :-)

Kind regards,
Dmjpro.

[r035198x]

Sorry for less time in my hand to look after your code throughly :-)
Have a look at my code ....

[code=java]
BufferedReader in = new BufferedReader( new FileReader("src _file"));
FileWriter out = new FileWriter("tgt _file");
String line;
HashSet hs = new HashSet();
String op = "";
while(!(line=in .readLine())!=n ull){
StringTokenizer s = new StringTokenizer (line,",");
String city;
while(s.hasMore Tokes()) city = s.nextToken();
op += hs.add(city) ? (line+System.ge tProperty("line .separator")) : "";
//here it ignores duplicate items
}
out.write(op);
in.close();
op.close();
[/code]

Enjoy this code.
Good Luck :-)

Kind regards,
Dmjpro.

@OP Do not use InputStreams to read text files. Use FileReader or Scanner
@dmjpro Do not use StringTokenizer

[JosAH]

@OP: what should be done with the following file contents:

1 2 x
3 4 x
5 6 x

or with this?

1 2 x
3 4 y
5 6 x

I think the problem description should be less ambiguous and more complete
to start with before we can start thinking of a solution.

kind regards,

Jos

[topala1984]

more info

more info

input :
1 2 x
3 4 x
5 6 y

output :
1 2 x
5 6 y

correct:
1 4 x
5 6 y

compare the third element of line 1 with the third element of line 2.
if equal join first element from line 1 with second element from line 2 and the third comun element in ... all in new file
if not equal put line 2 in new file

compare the third element of line 2 with the third element of line 3.
if equal join first element from line 2 with second element from line 3 and the third comun element ... all in new file
if not equal put line 3 in new file
............... ..............

compare the third element of line line n-1 with the third element of line n.
if equal join first element from line n-1 with second element from line n and the third comun element ... all in new file
if not equal put line n in new file

tnx all

[dmjpro]

more info

input :
1 2 x
3 4 x
5 6 y

output :
1 2 x
5 6 y

correct:
1 4 x
5 6 y

compare the third element of line 1 with the third element of line 2.
if equal join first element from line 1 with second element from line 2 and the third comun element in ... all in new file
if not equal put line 2 in new file

compare the third element of line 2 with the third element of line 3.
if equal join first element from line 2 with second element from line 3 and the third comun element ... all in new file
if not equal put line 3 in new file
............... ..............

compare the third element of line line n-1 with the third element of line n.
if equal join first element from line n-1 with second element from line n and the third comun element ... all in new file
if not equal put line n in new file

tnx all

Did you try my code :-)

Debasis Jana

[JosAH]

Did you try my code :-)

Debasis Jana

I noticed that you're spoonfeeding quite a bit of (incorrect) code lately; please don't do that.

Jos