Removing duplicates

[ravis]

Hi All,

I have a file which contains dupilcate records,i need to remove the duplicates and have to create a new file using perl programming.Can anybody guide me how can i do that as i m a newbie in perl programming.

Thanks

[KevinADC]

Hi All,

I have a file which contains dupilcate records,i need to remove the duplicates and have to create a new file using perl programming.Can anybody guide me how can i do that as i m a newbie in perl programming.

Thanks

you want to look into:

open() function
"while" loop
"print" command
close() function

http://perldoc.perl.org/

http://www.perl.org/books/beginning-perl/

[ravis]

My question is how can i do that using any perl function to check for duplicated or i need to loop complete file and check each record.
I can't read each record as it contains more a million records.

you want to look into:

open() function
"while" loop
"print" command
close() function

http://perldoc.perl.org/

http://www.perl.org/books/beginning-perl/

[KevinADC]

You obviously have to loop through the entire file to find all duplicates. I have a barebones snippet that does what you are asking about:

#!/usr/bin/perl

use strict;
use warnings;

my $file = '/path/to/file.txt';
my $output = 'path/to/outfile.txt';
open (OUT, ">" , $output) or die "$!";
my %seen = ();
{
   local @ARGV = ($file);
   local $^I = '.bac';
   while(<>){
      $seen{$_}++;
      next if $seen{$_} > 1;
      print OUT;
   }
}
print "finished processing file.";

this will create a backup copy of the original file but that can be deleted after the code runs. May take a while to run with that many lines in the file.