log parsing: gaps between transactions

[Arthur Dent]

I have a log file in this format:

YYYYMMDD HHMMSS DATADATADATADAT ADATA RESULTCODE

and I want to be able to parse the log and gleen information about gaps in
time between transactions. For example, it would be nice to know that 8
times in the past 24 hours we had periods over 5 minutes where no
transactions were successful. In order to do that, I need to compare each
line against the next based on result code and time elapsed. To make this
a bit more tricky, not every line in the log is in this format (approx 45%
are) but I only want to parse the lines that are. To add another problem,
I am an inexperienced perl programmer. If someone can point me in the
right direction it would be most appreciated. I have all of the O'Reilly
books and have referenced them but nothing seems to be close to what I
need. I also did an extensive search on deja before posting.

Thanks,

Arthur

[Jim Gibson]

Re: log parsing: gaps between transactions

In article <qn8ihvdw8iyp.s 3q484ucmkd5$.dl g@40tude.net>, Arthur Dent
<adent@fatadmin .com> wrote:
[color=blue]
> I have a log file in this format:
>
> YYYYMMDD HHMMSS DATADATADATADAT ADATA RESULTCODE
>
> and I want to be able to parse the log and gleen information about gaps in
> time between transactions. For example, it would be nice to know that 8
> times in the past 24 hours we had periods over 5 minutes where no
> transactions were successful. In order to do that, I need to compare each
> line against the next based on result code and time elapsed. To make this
> a bit more tricky, not every line in the log is in this format (approx 45%
> are) but I only want to parse the lines that are. To add another problem,
> I am an inexperienced perl programmer. If someone can point me in the
> right direction it would be most appreciated. I have all of the O'Reilly
> books and have referenced them but nothing seems to be close to what I
> need. I also did an extensive search on deja before posting.
>
> Thanks,
>
> Arthur[/color]

Have you tried any code? Do you have access to the perl online
documentation (try "perldoc perl" at a command line)? Check out
'perldoc perlquickre' for regular expressions and 'perldoc perlfaq5'
for file I/O.

Do you know how to read text files? You can identify the transaction
lines and extract the time information with a regular expression such
as

/(\d{8})\s+(\d\d )(\d\d)(\d\d)/

This looks for and saves an 8-digit date, followed by some white space,
followed by 3 sets of 2-digit values. If your times are in 24-hour
format, you can compute a time-since-midnight easily. Here is a sample
program:

#!/opt/perl/bin/perl

use strict;
use warnings;

my $prev = 86400;
while(<DATA>) {
if( /(\d{8})\s+(\d\d )(\d\d)(\d\d)/ ) {
my $date = $1;
my $time = $4 + 60*( $3 + 60*$2);
my $gap = $time - $prev;
if( $gap > 300 ) {
print "date = $date, time = $time, prev = $prev, gap = $gap
seconds\n";
}
$prev = $time;
}
}

__DATA__
line 1
line 2
20031127 123456 data1 code1
line 3
line 4
20031127 123457 data2 code2
line 5
line 6
20031127 123958 data3 code3
line 7
line 8
20031127 124000 data4 code4
line 9
line 10
line 11

__END__


Output:
date = 20031127, time = 45598, prev = 45297, gap = 301 seconds

You will have to modify this program to handle the case around
midnight, but that shouldn't be too hard. If you need more accurate,
absolute time, check out the Time::ParseDate module from

http://www.cpan.org/

FYI: this newsgroup is defunct. Try comp.lang.perl. misc in the future,
paying attention to the guidelines for that group, available at

http://mail.augustmail.com/~tadmc/clpmisc/clpmisc_guidelines.html