Remove duplicate entries from TXT file ?

[m|sf|t]

All,
Is it possible to use PHP to open/read a TXT file (i.e. IP.TXT) that
contains ip addresses (1 per line), remove any duplicates found, and
re-write the file back out to IP_NEW.TXT ?

The reason is, I have ips coming from several sources, i.e. URLSCAN logs,
MySQL dbs, etc. and I can use a DISTINCT on the individual queries, but I
then combine all the results into 1 file which may contain duplicates.

Thanks.

[Christopher Finke]

Re: Remove duplicate entries from TXT file ?

"m|sf|t" <m|sf|t@ampsych o.com> wrote in message
news:10um8se23r crf43@corp.supe rnews.com...[color=blue]
> Is it possible to use PHP to open/read a TXT file (i.e. IP.TXT) that
> contains ip addresses (1 per line), remove any duplicates found, and
> re-write the file back out to IP_NEW.TXT ?[/color]

Something like this...

<?php

$handle = fopen("ip.txt") ;
$file_data = file($handle);
$file_data = array_unique($f ile_data);
fclose($handle) ;

$new_handle = fopen("ip_new.t xt","w");
$new_file_data = implode("\n",$f ile_data);
fwrite($new_han dle, $new_file_data) ;
fclose($new_han dle);

?>

I'm pretty sure this works, but correct me if I'm wrong.

Chris Finke

[m|sf|t]

Re: Remove duplicate entries from TXT file ?

> Something like this...[color=blue]
>
> <?php
>
> $handle = fopen("ip.txt") ;
> $file_data = file($handle);
> $file_data = array_unique($f ile_data);
> fclose($handle) ;
>
> $new_handle = fopen("ip_new.t xt","w");
> $new_file_data = implode("\n",$f ile_data);
> fwrite($new_han dle, $new_file_data) ;
> fclose($new_han dle);
>
> ?>[/color]

Many thanks - I will give yours a try. A helluva lot simpler than the
copy/pasting I did from the PHP.NET manuals. This is what I came up with
(don't laugh at how complex it is ;)

<?php

function my_array_unique ($somearray)
{
$tmparr = array_unique($s omearray);
$i = 0;
foreach ($tmparr as $v) {
$newarr[$i] = $v;
$i++;
}
return $newarr;
}

function remdupe ($filename, $return_max_lin es = 0, $callback_func = null,
$do_rtrim = true, $buffer_size = 1024)
{
$open = fopen($filename , 'rb');
$open_data = array();
$line = 0;
while (!feof($open)) {
if ($do_rtrim) {
$open_data[$line] = rtrim(fgets($op en, $buffer_size));
} else {
$open_data[$line] = fgets($open, $buffer_size);
}

if ($callback_func != null && function_exists ($callback_func )) {
eval($callback_ func . '($open_data[$line]);');
}
$line++;

if ($return_max_li nes > 0) {
if ($line >= $return_max_lin es) {
break;
}
}
}
fclose($open);
return $open_data;
}

$open = remdupe("ip_com bined.txt");
$open = my_array_unique ($open);

$fp = @fopen("ip.txt" , "w");
$numElements = count($open);

for($counter = 0; $counter < $numElements - 1; $counter++) {
fwrite($fp, $open[$counter][0] . "\r\n");
}
fclose($fp);

?>

[Tim Van Wassenhove]

Re: Remove duplicate entries from TXT file ?

On 2005-01-17, m|sf|t <m|sf|t@ampsych o.com> wrote:[color=blue]
> All,
> Is it possible to use PHP to open/read a TXT file (i.e. IP.TXT) that
> contains ip addresses (1 per line), remove any duplicates found, and
> re-write the file back out to IP_NEW.TXT ?[/color]

On a linux system

timvw@madoka:~$ cat ip.txt
200.12.123.45
134.58.127.1
134.58.126.3
134.58.127.1
timvw@madoka:~$ sort ip.txt | uniq
134.58.126.3
134.58.127.1
200.12.123.45

--
Met vriendelijke groeten,
Tim Van Wassenhove <http://www.timvw.info>