File Comparsision

[ksrashmi]

HI All,

I want compare 2 flat files which contains data from database .
i should compare the old file with new files and want to create 3 files
which are deleted , added , modified

delete should contain the data which are not there in New File

added should contain the data which are not there in Old file

modified should contain which are values modified in new file .

i have written code for this using java it works for smaller files . but as the file size increase it is giving me Out of Memory exception .

i tried by increasing the Jvm memory . but nothing is working can any one tell me any other way .

i am using ArrayLIst to store the values


Thanks in advance

[r035198x]

HI All,

I want compare 2 flat files which contains data from database .
i should compare the old file with new files and want to create 3 files
which are deleted , added , modified

delete should contain the data which are not there in New File

added should contain the data which are not there in Old file

modified should contain which are values modified in new file .

i have written code for this using java it works for smaller files . but as the file size increase it is giving me Out of Memory exception .

i tried by increasing the Jvm memory . but nothing is working can any one tell me any other way .

i am using ArrayLIst to store the values


Thanks in advance

What size of files do you start getting problems at? Approx how many lines?

[ksrashmi]

What size of files do you start getting problems at? Approx how many lines?

around 150 MB files are giving problem . 1 MB files contains 1800 lines .. so it will be approximate 2,70,000 lines

[r035198x]

around 150 MB files are giving problem . 1 MB files contains 1800 lines .. so it will be approximate 2,70,000 lines

Well 150mb file is pretty big. If you don't get a nicer solution you may end up trying splitting the files first into smaller files and comparing the smaller files. This will require a lot of work.

Can't the comparisons be done using SQL at the database side?

[blazedaces]

HI All,

I want compare 2 flat files which contains data from database .
i should compare the old file with new files and want to create 3 files
which are deleted , added , modified

delete should contain the data which are not there in New File

added should contain the data which are not there in Old file

modified should contain which are values modified in new file .

i have written code for this using java it works for smaller files . but as the file size increase it is giving me Out of Memory exception .

i tried by increasing the Jvm memory . but nothing is working can any one tell me any other way .

i am using ArrayLIst to store the values


Thanks in advance

What do you need to do with these values? Have you considered not storing them in array lists? Have you considered writing the values in some way to a file which can be looked at later by another program in a memory-efficient way as well?

Just a suggestion, I'm dealing with a similar problem now and that's my solutions anyway...

Good luck,
-blazed

[ksrashmi]

actaully i am not sure about what is the use of those files . it was client requirement ..

we need to compare 2 files and send deleted, added and modified file .

since i need to compare files i need to store one file value in the List and other file i can read then i can compare so i used ArrayLIst store the values .

What do you need to do with these values? Have you considered not storing them in array lists? Have you considered writing the values in some way to a file which can be looked at later by another program in a memory-efficient way as well?

Just a suggestion, I'm dealing with a similar problem now and that's my solutions anyway...

Good luck,
-blazed

[blazedaces]

actaully i am not sure about what is the use of those files . it was client requirement ..

we need to compare 2 files and send deleted, added and modified file .

since i need to compare files i need to store one file value in the List and other file i can read then i can compare so i used ArrayLIst store the values .

My question is then how many values in each file to you need to "compare" at one go? Like, are their 1000 values, each of which you have to compare only to the other corresponding value? If so, compare the first pair, print to a file your "synopsis or conclusion or comparison information" and simply move onto the next pair. This way you won't be trying to store 300mb or more in Java. You could even program it to print out some kind of table or excel or whatever your client might prefer. I mean, how does your client want to receive the conclusive comparison info?

Good luck,
-blazed

[ksrashmi]

My question is then how many values in each file to you need to "compare" at one go? Like, are their 1000 values, each of which you have to compare only to the other corresponding value? If so, compare the first pair, print to a file your "synopsis or conclusion or comparison information" and simply move onto the next pair. This way you won't be trying to store 300mb or more in Java. You could even program it to print out some kind of table or excel or whatever your client might prefer. I mean, how does your client want to receive the conclusive comparison info?

Good luck,
-blazed

we are having 33 column in each line .. i am combining the 2 column and making it as key and all other field are converted into hashcode and stored ..
if there is change in any of the fields we need to store it as modified . so i am checking the old hashcode value and new hashcode if there is any change in the hashcode it is stored as modified


yes i need to compare at one go because in the Old file it may be at the 1 st line and New file it may be in last line . i mean line number for same data will be different . so i can't split the file into smaller files .


i need to compare files upto 20GB are there any tools or is there any way to do this pls help me

[JosAH]

Can you sort those files according to the PKs in the database? If so, you can
simply walk through the old and new file and compare single lines from both files:

1) if present in the old file but not in the new file it's deleted;
2) if present in the new file but not in the old file it's inserted;
3) if anything except the PK was changed it's modified.

kind regards,

Jos

[Bharanidharan]

Can you sort those files according to the PKs in the database? If so, you can
simply walk through the old and new file and compare single lines from both files:

1) if present in the old file but not in the new file it's deleted;
2) if present in the new file but not in the old file it's inserted;
3) if anything except the PK was changed it's modified.

kind regards,

Jos

If you show the piece of code you wrote i can help you on that. You should derefer the unnecessary objects at the appropriate time and use garbage collection concepts efficiently.

[JosAH]

If you show the piece of code you wrote i can help you on that. You should derefer the unnecessary objects at the appropriate time and use garbage collection concepts efficiently.

Erm, I'm afraid you replied to the wrong person; I'm not the OP and I'm not going
to show you any code. I was just helping out the OP too ;-)

kind regards,

Jos