regular expression inquiry

[vito]

i've a text file which stores thousands of records.

now i find that there is reduncdant records and would like to remove them.

e.g.

abc

he is a boy.
he lives in smallvilli
he has 10 bros

def

he has a dad

abc

he is a boy.
he lives in smallvilli
he has 10 bros

abc

he is a boy.
he lives in bigcity


the 1st & 3rd are duplicates but the 4th one needs some attention.

is there any convenient function to streamline this checking procedure?
thanks in advance

[Janwillem Borleffs]

Re: regular expression inquiry

vito wrote:

the 1st & 3rd are duplicates but the 4th one needs some attention.
>
is there any convenient function to streamline this checking
procedure? thanks in advance
>

Duplicates can be removed, e.g. with the following (when the text file isn't
too big!):

$file = file('test.txt' );
$unique = array_unique($f ile);
$fp = fopen('test2.tx t', 'w');
fputs($fp, implode('', $unique));
fclose($fp);

Where test2.txt now contains the unique records only. To detect the lines
that needs attention, you will need to define the criteria how to detect
them.


JW

[Alvaro G. Vicario]

Re: regular expression inquiry

*** Janwillem Borleffs escribió/wrote (Sat, 15 Jul 2006 10:39:17 +0200):

Duplicates can be removed, e.g. with the following (when the text file isn't
too big!):

For big files, I suppose you can store records in a hash table (database or
array) where the key is a hash or checksum of the record content, such as
MD5.


--
-+ http://alvaro.es - Álvaro G. Vicario - Burgos, Spain
++ Mi sitio sobre programación web: http://bits.demogracia.com
+- Mi web de humor con rayos UVA: http://www.demogracia.com
--