Parsing log files

cajun has asked for the wisdom of the Perl Monks concerning the following question:

I'm trying to parse some log files and print reports for each day. The problem is the current way I'm detecting a new day, the very last day in the log file never gets printed. I'm sure I'm overlooking an incredibly simple solution, but I've not found it yet.

#!/usr/bin/perl -w

use strict;

my $date;
my $lastdate = '';

while(<DATA>){
   $date = $_;
   if (($lastdate ne $date) && ($lastdate ne '')){
      # print $lastdate report here
      print "$.    $_";
   }
   $lastdate = $date;
}
__DATA__
10
10
10
11
11
11
12
12
12
[download]

Update Of course the snippet posted above is an over simplification of the problem. Thanks to all for the ideas.

The solution was given in mifflin's code but what drove it home was when GrandFather said "The bug fix is to let the while loop run one more itteration after finding the end of the file."

I'm parsing maillog files to produce a SpamAssassin report. I foolishly started with someone elses code that did not work, did not use strict, did not use warnings.. I would have been better off I think to do the whole thing from scratch.

Comment on Parsing log files Download Code

Replies are listed 'Best First'.
Re: Parsing log files by mifflin (Curate) on Jun 28, 2005 at 03:01 UTC
Here's my solution... `use warnings; use strict; my $date; my $lastdate = ''; my $line = 0; while(<DATA>){ $line++; $date = $_; if (($lastdate ne $date) && ($lastdate ne '')){ print "$line: $lastdate"; } $lastdate = $date; } print "$line: $lastdate"; <>; __DATA__ 10 10 10 11 11 11 12 12 12` [download] produces... `4: 10 7: 11 9: 12` [download]	[reply] [d/l] [select]
Re: Parsing log files by GrandFather (Saint) on Jun 28, 2005 at 03:33 UTC
The following variant uses the "return the true value" property of \|\| to set a variable if it is currently false. It also uses next with a modifier to reduce nesting by a level. The bug fix is to let the while loop run one more itteration after finding the end of the file. `use strict; my $date; my $lastdate; while (($date = <DATA>) or $lastdate){ $lastdate \|\|= $date; next if $lastdate eq $date; # print $lastdate report here print "Line $. ends date $lastdate"; $lastdate = $date; }` [download] Read more... OP's data and run result (303 Bytes) Update: add bug fix explanation Perl is Huffman encoded by design.	[reply] [d/l] [select]
Re: Parsing log files by tlm (Prior) on Jun 28, 2005 at 03:02 UTC
I'd do something like this: `my $date; my $lastdate = ''; my $logdata = ''; while ( <LOGFILE> ) { $date = extract_date( $_ ); $logdata .= $stuff; if (length $lastdate and $lastdate ne $date) { print_report( $lastdate, $logdata ); $logdata = ''; } $lastdate = $date; } print_report( $lastdate, $logdata );` [download] That's the rough idea, anyway. Instead of `extract_date` you may have a `parse` method that returns a date and whatever else you want from each line. E.g.: `( $date, my $stuff ) = parse( $_ ); $logdata .= $stuff; # etc.` [download] Also, barring some crazy scheme for the date, the test in the loop could be simplified to `if ( $lastdate and $lastdate ne $date ) {` [download] Another possibility, if you want to keep everything within the loop, is to use `eof` to test for the end-of-file condition. Then the above would become: `my $date; my $lastdate = ''; my $logdata = ''; while ( <LOGFILE> ) { ( $date, my $stuff ) = parse( $_ ); $logdata .= $stuff; if ( $lastdate and ( $lastdate ne $date or eof LOGFILES ) ) { print_report( $lastdate, $logdata ); $logdata = ''; } $lastdate = $date; }` [download] the lowliest monk	[reply] [d/l] [select]


Come for the quick hacks, stay for the epiphanies.
	PerlMonks