Optimizing a string manipulation script.

[Cleverbum@hotmail.com]

I'm not really accustomed to string manipulation and so I was wondering
if any of you could be any help i speeding up this script intended to
change the format of some saved log information into a CSV file while
removing duplicate records.
The main problem is that the script currently takes about 20 seconds to
execute, and were it to take much longer it would time out.

Below is the script itself, and then some example lines from the log
file it processes:

<?
$data ="";

$fp = fopen("logs.txt ", "r");

while(!feof($fp )){
$data .= fread($fp, 4096);
}
fclose($fp);

$fullArray = explode("\n", $data);
$ArrayofArrays[0] = array("","");
$myArray[0]=$fullArray[0];

(int)$flg;
for($i=0;$i<cou nt($fullArray); $i++){
$flg=0;
for($j=0;$j<cou nt($myArray);$j ++){
if($myArray[$j]==$fullArray[$i]){$flg++;}
}
if($flg==0){
$myArray[count($myArray)]=$fullArray[$i];
}
}

for($maincount= 0;$maincount<co unt($myArray);$ maincount++){
$newArray = explode("\"",$m yArray[$maincount]);
$newArray[0] = str_replace(arr ay("[","]","+"),"", $newArray[0]);
$outArray = explode(" ", $newArray[0]);


$tmpArray = explode(" ", $newArray[1]);
$j=count($outAr ray);
for($i=$j;$i<$j +count($tmpArra y);$i++){
$outArray[$i] = $tmpArray[$i-$j];
}

$tmpArray = explode(" ", $newArray[2]);
$j=count($outAr ray);
for($i=$j;$i<$j +count($tmpArra y);$i++){
$outArray[$i] = $tmpArray[$i-$j];
}
$outArray[count($outArray )] = $newArray[3];
$outArray[count($outArray )] = $newArray[5];

trim_array($out Array, " \n\t:;,");
$ArrayofArrays[$maincount]=$outArray;
}

$out = fopen("output.c sv", "a");
for($i=0;$i<cou nt($ArrayofArra ys);$i++){
for($j=0;$j<cou nt($ArrayofArra ys[$i]);$j++){
if($ArrayofArra ys[$i][$j]!=""){
fwrite($out,$Ar rayofArrays[$i][$j]);
fwrite($out,"," );
}
}
fwrite($out,"\n ");
}
fclose($out);

//printout($Final AofAs);

function printout($a){
echo "<br><br>";
for($i=0;$i<cou nt($a);$i++){
if(count($a[$i])!=1 && count($a[$i])!=0){printout( $a[$i]);}
else{
if($a[$i]!="" && $a[$i]!="-"){
echo $a[$i];
echo "<br>";
}
}
}
}
function trim_array($a /*array to be trimmed*/,$b /*string of chars to
be removed*/){
for($i=0;$i<cou nt($a);$i++){
$a[$i]=trim($a[$i],$b);
}
}

?>


jpgme.co.uk: [25/May/2006:13:04:47 +0100] 81.157.187.150 - - "GET /
HTTP/1.1" 200 5736 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT
5.1; SV1; .NET CLR 1.1.4322)"
jpgme.co.uk: [25/May/2006:13:04:47 +0100] 81.157.187.150 - - "GET /
HTTP/1.1" 200 5736 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT
5.1; SV1; .NET CLR 1.1.4322)"
jpgme.co.uk: [25/May/2006:13:04:47 +0100] 81.157.187.150 - - "GET
/images/title.gif HTTP/1.1" 200 5237 "http://www.martinsphot os.co.uk/"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR
1.1.4322)"
jpgme.co.uk: [25/May/2006:13:04:48 +0100] 81.157.187.150 - - "GET
/images/but1.gif HTTP/1.1" 200 696 "http://www.martinsphot os.co.uk/"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR
1.1.4322)"
jpgme.co.uk: [25/May/2006:13:04:48 +0100] 81.157.187.150 - - "GET
/images/vline.gif HTTP/1.1" 200 85 "http://www.martinsphot os.co.uk/"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR
1.1.4322)"
jpgme.co.uk: [25/May/2006:13:04:48 +0100] 81.157.187.150 - - "GET
/images/but2.gif HTTP/1.1" 200 742 "http://www.martinsphot os.co.uk/"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR
1.1.4322)"
jpgme.co.uk: [25/May/2006:13:04:48 +0100] 81.157.187.150 - - "GET
/images/but3.gif HTTP/1.1" 200 742 "http://www.martinsphot os.co.uk/"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR
1.1.4322)"
jpgme.co.uk: [25/May/2006:13:04:48 +0100] 81.157.187.150 - - "GET
/images/but4.gif HTTP/1.1" 200 506 "http://www.martinsphot os.co.uk/"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR
1.1.4322)"
jpgme.co.uk: [25/May/2006:13:04:48 +0100] 81.157.187.150 - - "GET
/images/but5.gif HTTP/1.1" 200 711 "http://www.martinsphot os.co.uk/"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR
1.1.4322)"
jpgme.co.uk: [25/May/2006:13:04:48 +0100] 81.157.187.150 - - "GET
/images/but6.gif HTTP/1.1" 200 600 "http://www.martinsphot os.co.uk/"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR
1.1.4322)"
jpgme.co.uk: [25/May/2006:13:04:48 +0100] 81.157.187.150 - - "GET
/images/ind_th2.jpg HTTP/1.1" 200 29533
"http://www.martinsphot os.co.uk/" "Mozilla/4.0 (compatible; MSIE 6.0;
Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
jpgme.co.uk: [25/May/2006:13:04:48 +0100] 81.157.187.150 - - "GET
/images/ind_th1.jpg HTTP/1.1" 200 18673
"http://www.martinsphot os.co.uk/" "Mozilla/4.0 (compatible; MSIE 6.0;
Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
jpgme.co.uk: [25/May/2006:13:04:49 +0100] 81.157.187.150 - - "GET
/images/ind_th3.jpg HTTP/1.1" 200 9298
"http://www.martinsphot os.co.uk/" "Mozilla/4.0 (compatible; MSIE 6.0;
Windows NT 5.1; SV1; .NET CLR 1.1.4322)"
jpgme.co.uk: [25/May/2006:13:04:54 +0100] 81.157.187.150 - - "GET
/gallery.php HTTP/1.1" 200 5787 "http://www.martinsphot os.co.uk/"
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR
1.1.4322)"

[Alan Little]

Re: Optimizing a string manipulation script.

Carved in mystic runes upon the very living rock, the last words of
<Cleverbum@hotm ail.com> of comp.lang.php make plain:
[color=blue]
> I'm not really accustomed to string manipulation and so I was wondering
> if any of you could be any help i speeding up this script intended to
> change the format of some saved log information into a CSV file while
> removing duplicate records.
> The main problem is that the script currently takes about 20 seconds to
> execute, and were it to take much longer it would time out.
>
> Below is the script itself, and then some example lines from the log
> file it processes:
>
> <?
> [snip]
> ?>[/color]

Whew!

How about an example of the output you're trying to achieve? That might
be easier.

--
Alan Little
Phorm PHP Form Processor

http://www.phorm.com/

[Cleverbum@hotmail.com]

Re: Optimizing a string manipulation script.


Alan Little wrote:
[color=blue]
> Carved in mystic runes upon the very living rock, the last words of
> <Cleverbum@hotm ail.com> of comp.lang.php make plain:
>[color=green]
> > I'm not really accustomed to string manipulation and so I was wondering
> > if any of you could be any help i speeding up this script intended to
> > change the format of some saved log information into a CSV file while
> > removing duplicate records.
> > The main problem is that the script currently takes about 20 seconds to
> > execute, and were it to take much longer it would time out.
> >
> > Below is the script itself, and then some example lines from the log
> > file it processes:
> >
> > <?
> > [snip]
> > ?>[/color]
>
> Whew!
>
> How about an example of the output you're trying to achieve? That might
> be easier.
>
> --
> Alan Little
> Phorm PHP Form Processor
> http://www.phorm.com/[/color]

Here we go then:

jpgme.co.uk:,25/May/2006:13:04:47,0 100,81.157.187. 150,-,-,GET,/,HTTP/1.1,200,5736,-,Mozilla/4.0
(compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322),
jpgme.co.uk:,25/May/2006:13:04:47,0 100,81.157.187. 150,-,-,GET,/images/title.gif,HTTP/1.1,200,5237,http://www.martinsphotos.co.uk/,Mozilla/4.0
(compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322),
jpgme.co.uk:,25/May/2006:13:04:48,0 100,81.157.187. 150,-,-,GET,/images/but1.gif,HTTP/1.1,200,696,http://www.martinsphotos.co.uk/,Mozilla/4.0
(compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322),
jpgme.co.uk:,25/May/2006:13:04:48,0 100,81.157.187. 150,-,-,GET,/images/vline.gif,HTTP/1.1,200,85,http://www.martinsphotos.co.uk/,Mozilla/4.0
(compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322),

[Alan Little]

Re: Optimizing a string manipulation script.

Carved in mystic runes upon the very living rock, the last words of
<Cleverbum@hotm ail.com> of comp.lang.php make plain:
[color=blue]
> Alan Little wrote:
>[color=green]
>> Carved in mystic runes upon the very living rock, the last words of
>> <Cleverbum@hotm ail.com> of comp.lang.php make plain:
>>[color=darkred]
>> > I'm not really accustomed to string manipulation and so I was
>> > wondering if any of you could be any help i speeding up this script
>> > intended to change the format of some saved log information into a
>> > CSV file while removing duplicate records.
>> > The main problem is that the script currently takes about 20
>> > seconds to execute, and were it to take much longer it would time
>> > out.
>> >
>> > Below is the script itself, and then some example lines from the
>> > log file it processes:
>> >
>> > <?
>> > [snip]
>> > ?>[/color]
>>
>> Whew!
>>
>> How about an example of the output you're trying to achieve? That
>> might be easier.[/color]
>
> Here we go then:[/color]

Try this:

<?php
$patt =
'!([^:]+:) \[([^:]+:\d\d:\d\d:\d\ d) [+-](\d{4})\] '.
'(\d+\.\d+\.\d+ \.\d+) (-) (-) "(\w+) (/[^ ]*) '.
'(HTTP/\d\.\d)" (\d+) (\d+) "([^"]+)" "([^"]+)"'.
"\n?".'!';

$log = fopen('log.csv' , 'a');

$logfile = file_get_conten ts('logs.txt');
$logfile = ereg_replace("\ r\n?", "\n", $logfile);

preg_match_all( $patt, $x, $matches, PREG_SET_ORDER) ;

foreach($matche s as $match) {
unset($match[0]);
$logline = implode(',', $match);
fputs($log, $logline."\n");
}

fclose($log);
?>

I don't know what those two blank log elements are after the IP, so this
pattern will only work when they're blank.

--
Alan Little
Phorm PHP Form Processor

http://www.phorm.com/

[Stephen Kellett]

Re: Optimizing a string manipulation script.

In message <1149628457.141 962.238760@c74g 2000cwc.googleg roups.com>,
Cleverbum@hotma il.com writes[color=blue]
>I'm not really accustomed to string manipulation and so I was wondering
>if any of you could be any help i speeding up this script intended to
>change the format of some saved log information into a CSV file while
>removing duplicate records.
>The main problem is that the script currently takes about 20 seconds to
>execute, and were it to take much longer it would time out.[/color]

You may also want to look at PHP Performance Validator. This is a code
profiler for PHP. No requirement to modify your code. Works with PHP 4
and PHP 5. Its in beta at the moment. Windows only.

http://www.softwareverify.com/phpPerformanceValidator/index.html

Stephen
--
Stephen Kellett
Object Media Limited http://www.objmedia.demon.co.uk/software.html
Computer Consultancy, Software Development
Windows C++, Java, Assembler, Performance Analysis, Troubleshooting

[Cleverbum@hotmail.com]

Re: Optimizing a string manipulation script.


Alan Little wrote:[color=blue]
> Carved in mystic runes upon the very living rock, the last words of
> <Cleverbum@hotm ail.com> of comp.lang.php make plain:
>[color=green]
> > Alan Little wrote:
> >[color=darkred]
> >> Carved in mystic runes upon the very living rock, the last words of
> >> <Cleverbum@hotm ail.com> of comp.lang.php make plain:
> >>
> >> > I'm not really accustomed to string manipulation and so I was
> >> > wondering if any of you could be any help i speeding up this script
> >> > intended to change the format of some saved log information into a
> >> > CSV file while removing duplicate records.
> >> > The main problem is that the script currently takes about 20
> >> > seconds to execute, and were it to take much longer it would time
> >> > out.
> >> >
> >> > Below is the script itself, and then some example lines from the
> >> > log file it processes:
> >> >
> >> > <?
> >> > [snip]
> >> > ?>
> >>
> >> Whew!
> >>
> >> How about an example of the output you're trying to achieve? That
> >> might be easier.[/color]
> >
> > Here we go then:[/color]
>
> Try this:
>
> <?php
> $patt =
> '!([^:]+:) \[([^:]+:\d\d:\d\d:\d\ d) [+-](\d{4})\] '.
> '(\d+\.\d+\.\d+ \.\d+) (-) (-) "(\w+) (/[^ ]*) '.
> '(HTTP/\d\.\d)" (\d+) (\d+) "([^"]+)" "([^"]+)"'.
> "\n?".'!';
>
> $log = fopen('log.csv' , 'a');
>
> $logfile = file_get_conten ts('logs.txt');
> $logfile = ereg_replace("\ r\n?", "\n", $logfile);
>
> preg_match_all( $patt, $x, $matches, PREG_SET_ORDER) ;
>
> foreach($matche s as $match) {
> unset($match[0]);
> $logline = implode(',', $match);
> fputs($log, $logline."\n");
> }
>
> fclose($log);
> ?>
>
> I don't know what those two blank log elements are after the IP, so this
> pattern will only work when they're blank.
>
> --
> Alan Little
> Phorm PHP Form Processor
> http://www.phorm.com/[/color]

They do seem to stay blank for the entire log, which means it should be
fine. I noticed though that your version produced smaller files that
mine, and on closer inspection I noticed some log lines sent it a
little insane. It seems to have problems when there's no file size
sent, for example on lines with errors. The following log lines are the
ones causing problems:

jpgme.co.uk: [26/May/2006:10:12:38 +0100] 130.88.199.23 - - "GET
/addthumbs.php HTTP/1.1" 200 37 "-" "Mozilla/5.0 (X11; U; Linux i686;
en-US; rv:1.7.12) Gecko/20060210 Fedora/1.7.12-1.3.3.legacy"
jpgme.co.uk: [26/May/2006:10:14:05 +0100] 130.88.199.23 - - "GET
/bulk.php HTTP/1.1" 200 42792 "-" "Mozilla/5.0 (X11; U; Linux i686;
en-US; rv:1.7.12) Gecko/20060210 Fedora/1.7.12-1.3.3.legacy"
jpgme.co.uk: [26/May/2006:10:14:23 +0100] 130.88.199.23 - - "GET
/add.php?folder= ./All_work/Flowers HTTP/1.1" 200 21384
"http://www.martinsphot os.co.uk/bulk.php" "Mozilla/5.0 (X11; U; Linux
i686; en-US; rv:1.7.12) Gecko/20060210 Fedora/1.7.12-1.3.3.legacy"
jpgme.co.uk: [26/May/2006:10:16:54 +0100] 130.88.199.23 - - "GET
/add.php?folder= ./All_work/Flowers HTTP/1.1" 200 18309
"http://www.martinsphot os.co.uk/bulk.php" "Mozilla/5.0 (X11; U; Linux
i686; en-US; rv:1.7.12) Gecko/20060210 Fedora/1.7.12-1.3.3.legacy"
jpgme.co.uk: [26/May/2006:10:16:54 +0100] 130.88.199.23 - - "GET
/images/folder2.png HTTP/1.1" 304 -
"http://www.martinsphot os.co.uk/add.php?folder= ./All_work/Flowers"
"Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12) Gecko/20060210
Fedora/1.7.12-1.3.3.legacy"
jpgme.co.uk: [26/May/2006:10:16:59 +0100] 130.88.199.23 - - "POST
/do_add.php HTTP/1.1" 200 12746
"http://www.martinsphot os.co.uk/add.php?folder= ./All_work/Flowers"
"Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12) Gecko/20060210
Fedora/1.7.12-1.3.3.legacy"
jpgme.co.uk: [26/May/2006:10:16:59 +0100] 130.88.199.23 - - "GET
/addone.php?alb= 61&folder=./All_work/Flowers&file=DS C04827.jpg
HTTP/1.1" 200 23231 "http://www.martinsphot os.co.uk/do_add.php"
"Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12) Gecko/20060210
Fedora/1.7.12-1.3.3.legacy"
jpgme.co.uk: [26/May/2006:10:16:59 +0100] 130.88.199.23 - - "GET
/addone.php?alb= 61&folder=./All_work/Flowers&file=DS C04822_1.JPG
HTTP/1.1" 200 23231 "http://www.martinsphot os.co.uk/do_add.php"
"Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12) Gecko/20060210
Fedora/1.7.12-1.3.3.legacy"
jpgme.co.uk: [26/May/2006:10:22:52 +0100] 130.88.199.23 - - "GET
/addone.php?alb= 61&folder=./All_work/Flowers&file=no rmal_DSC04962.j pg
HTTP/1.1" 200 23231 "http://www.martinsphot os.co.uk/do_add.php"
"Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12) Gecko/20060210
Fedora/1.7.12-1.3.3.legacy"
jpgme.co.uk: [26/May/2006:10:22:52 +0100] 130.88.199.23 - - "GET
/addone.php?alb= 61&folder=./All_work/Flowers&file=no rmal_DSC04980.j pg
HTTP/1.1" 200 23231 "http://www.martinsphot os.co.uk/do_add.php"
"Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12) Gecko/20060210
Fedora/1.7.12-1.3.3.legacy"
jpgme.co.uk: [26/May/2006:10:22:52 +0100] 130.88.199.23 - - "GET
/addone.php?alb= 61&folder=./All_work/Flowers&file=no rmal_DSC05000.j pg
HTTP/1.1" 200 20060 "http://www.martinsphot os.co.uk/do_add.php"
"Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12) Gecko/20060210
Fedora/1.7.12-1.3.3.legacy"
jpgme.co.uk: [26/May/2006:10:23:00 +0100] 130.88.199.23 - - "GET
/add.php?folder= ./All_work/Flowers HTTP/1.1" 200 22952
"http://www.martinsphot os.co.uk/bulk.php" "Mozilla/5.0 (X11; U; Linux
i686; en-US; rv:1.7.12) Gecko/20060210 Fedora/1.7.12-1.3.3.legacy"
jpgme.co.uk: [26/May/2006:10:23:00 +0100] 130.88.199.23 - - "GET
/images/folder2.png HTTP/1.1" 304 -
"http://www.martinsphot os.co.uk/add.php?folder= ./All_work/Flowers"
"Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12) Gecko/20060210
Fedora/1.7.12-1.3.3.legacy"
jpgme.co.uk: [26/May/2006:10:25:50 +0100] 130.88.199.23 - - "GET
/add.php?folder= ./All_work/Flowers HTTP/1.1" 200 18309
"http://www.martinsphot os.co.uk/bulk.php" "Mozilla/5.0 (X11; U; Linux
i686; en-US; rv:1.7.12) Gecko/20060210 Fedora/1.7.12-1.3.3.legacy"
jpgme.co.uk: [26/May/2006:10:25:50 +0100] 130.88.199.23 - - "GET
/images/folder2.png HTTP/1.1" 304 -
"http://www.martinsphot os.co.uk/add.php?folder= ./All_work/Flowers"
"Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12) Gecko/20060210
Fedora/1.7.12-1.3.3.legacy"