log file split into HoA

vboy1997 has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks,

I have a huge mail log file with each entry looking like this,

Dec 8 08:49:21 b.mx.sonic.net sm-mta18242: jB8GnCuK018242: from=<cj@oreilly.com>, size=10731, class="0", nrcpts=2, msgid=<E4461FEB-1B74-4612-80DC-3A39B04D89B1@oreilly.com>, proto=ESMTP, daemon=MTA, relay=mwest.oreilly.com 209.204.146.24
Dec 8 08:49:21 b.mx.sonic.net sm-mta18528: jB8GnCuK018242: to=<mkirk@corp.sonic.net>,<dane@corp.sonic.net>, delay=00:00:00, xdelay=00:00:00, mailer=esmtp, pri=160731, relay=lds.sonic.net. 208.201.249.231, dsn=2.0.0, stat=Sent (jB8GnLpC004736 Message accepted for delivery)

I want to split each entry into 4 parts and store them inside a HoA. The 4 parts are:

$header = everything before the $id
$id = jB8GnCuK018242
$to_from = from=<cj@oreilly.com> or to=<mkirk@corp.sonic.net>,<dane@corp.sonic.net>
$footer = everything after the $to_from section
Any help or pointers on how I can do this would be greatly appreciated. Thank you all.

Comment on log file split into HoA

Replies are listed 'Best First'.
Re: log file split into HoA by Anonymous Monk on May 15, 2011 at 06:47 UTC
Towards what end? There is probably a cpan module to parse it, perhaps Parse::Syslog::Mail	[reply]
Re: log file split into HoA by John M. Dlugosz (Monsignor) on May 15, 2011 at 05:38 UTC
Try using a regular expression with 4 captures. Your first task is to look carefully at the lines and figure out how to separate the parts. (Hint: ':' characters?).	[reply]
Re: log file split into HoA by LanX (Saint) on May 15, 2011 at 11:23 UTC
use strict; use warnings; use Data::Dump qw"pp"; use English; my %HoA; my @AoA; while (<DATA>) { if (my ($id,$fromto)=/: (\w+): ((?:from\|to)=<.?>),/ ){ $HoA{$id}=[$PREMATCH,$fromto,$POSTMATCH]; @AoA[$.]=[$PREMATCH,$id,$fromto,$POSTMATCH]; }else { warn "Couldn't parse <<<$_>>> at line $."; } } pp(\%HoA); pp(\@AoA); __DATA__ Dec 8 08:49:21 b.mx.sonic.net sm-mta18242: jB8GnCuK018242: from=<cj@or +eilly.com>, size=10731, class="0", nrcpts=2, msgid=<E4461FEB-1B74-461 +2-80DC-3A39B04D89B1@oreilly.com>, proto=ESMTP, daemon=MTA, relay=mwes +t.oreilly.com 209.204.146.24 Dec 8 08:49:21 b.mx.sonic.net sm-mta18528: jB8GnCuK018242: to=<mkirk@c +orp.sonic.net>,<dane@corp.sonic.net>, delay=00:00:00, xdelay=00:00:00 +, mailer=esmtp, pri=160731, relay=lds.sonic.net. 208.201.249.231, dsn +=2.0.0, stat=Sent (jB8GnLpC004736 Message accepted for delivery) [download] both data-lines have the same ID so maybe you don't want a HoA but a AoA OUTPUT: { jB8GnCuK018242 => [ "Dec 8 08:49:21 b.mx.sonic.net sm-mta18528", "to=<mkirk\@corp.sonic.net>", "<dane\@corp.sonic.net>, delay=00:00:00, xdelay=00:00:00, mail +er=esmtp, pri=160731, relay=lds.sonic.net. 208.201.249.231, dsn=2.0.0 +, stat=Sent (jB8GnLpC004736 Message accepted for delivery) \n", ], } [ undef, [ "Dec 8 08:49:21 b.mx.sonic.net sm-mta18242", "jB8GnCuK018242", "from=<cj\@oreilly.com>", " size=10731, class=\"0\", nrcpts=2, msgid=<E4461FEB-1B74-4612-80D +C-3A39B04D89B1\@oreilly.com>, proto=ESMTP, daemon=MTA, relay=mwest.or +eilly.com 209.204.146.24\n", ], [ "Dec 8 08:49:21 b.mx.sonic.net sm-mta18528", "jB8GnCuK018242", "to=<mkirk\@corp.sonic.net>", "<dane\@corp.sonic.net>, delay=00:00:00, xdelay=00:00:00, mailer=e +smtp, pri=160731, relay=lds.sonic.net. 208.201.249.231, dsn=2.0.0, st +at=Sent (jB8GnLpC004736 Message accepted for delivery) \n", ], ] [download] Cheers Rolf UPDATE: better add a chomp after the while ... UPDATE: adding a space to the regex helps catching additional mailadresses `my ($id,$fromto)=/: (\w+): ((?:from\|to)=<.?>), /;` but be aware that you have to be sure about the format of your logs, otherwise you need a parser for mailadresses which isn't trivial	[reply] [d/l] [select]


Pathologically Eclectic Rubbish Lister
	PerlMonks