Re: Re: Multi-Format Log Parser

You're right about the improper use of quotes within a user agent string. That could cause pattern matches to fail, and those would be skipped. I'm thinking about adding an option to print log lines that don't match the currently selected format to STDERR, or a count of lines that didn't match. From using this on a fairly large web site, I know the patterns match our traffic fairly well, but it will be interesting to see how many lines don't match and why. I did a dump of counts per unique user agent string using this log parser a few days ago for our QA department and in one day's worth of logs there were 82,279 unique user agent strings. Our QA guys are after percentages of traffic per browser and platform, and I don't relish their job of parsing all the user agent strings to get that information since they don't follow any standardized format.

Comment on Re: Re: Multi-Format Log Parser - Version 2.0

Replies are listed 'Best First'.

Re: Re: Re: Multi-Format Log Parser - Version 2.0
by cjensen (Sexton) on Jan 23, 2002 at 06:08 UTC

26c26
< GetOptions (\%optctl, "type|t=s", "pattern|p=s");
---
> GetOptions (\%optctl, "type|t=s", "pattern|p=s", "debug|d=i");
30,32c30,32
<   'common'   => [ qr{(\S+) (\S+) (\S+) \[([^\]]*)\] \"([^\"]*)\" (\d
++) (\d+)}, [qw(h l u t r c b)] ],
<   'virtual'  => [ qr{(\S+) (\S+) (\S+) (\S+) \[([^\]]*)\] \"([^\"]*)
+\" (\d+) (\d+)}, [qw(v h l u t r c b)] ],
<   'combined' => [ qr{(\S+) (\S+) (\S+) \[([^\]]*)\] \"([^\"]*)\" (\d
++) (\d+) \"([^\"]*)\" \"([^\"]*)\"}, [qw(h l u t r c b R A)] ],
---
>   'common'   => [ qr{(\S+) (\S+) (\S+) \[([^\]]*)\] \"([^\"]*)\" (\d
++) ([\d\-]+)}, [qw(h l u t r c b)] ],
>   'virtual'  => [ qr{(\S+) (\S+) (\S+) (\S+) \[([^\]]*)\] \"([^\"]*)
+\" (\d+) ([\d\-]+)}, [qw(v h l u t r c b)] ],
>   'combined' => [ qr{(\S+) (\S+) (\S+) \[([^\]]*)\] \"([^\"]*)\" (\d
++) ([\d\-]+) \"([^\"]*)\" \"([^\"]*)\"}, [qw(h l u t r c b R A)] ],
35,36c35,36
<   'extended' => [ qr{(\S+) (\S+) (\S+) \[([^\]]*)\] \"([^\"]*)\" (\d
++) (\d+) \"([^\"]*)\" \"([^\"]*)\" (\d+) (\d+)}, [qw(h l u t r c b R 
+A P T)] ],
<   'custom'   => [ qr{(\S+) (\S+) (\S+) \[([^\]]*)\] \"([^\"]*)\" (\d
++) (\d+) \"([^\"]*)\" \"([^\"]*)\" (\d+)}, [qw(h l u t r c b A R T)] 
+],
---
>   'extended' => [ qr{(\S+) (\S+) (\S+) \[([^\]]*)\] \"([^\"]*)\" (\d
++) ([\d\-]+) \"([^\"]*)\" \"([^\"]*)\" (\d+) (\d+)}, [qw(h l u t r c 
+b R A P T)] ],
>   'custom'   => [ qr{(\S+) (\S+) (\S+) \[([^\]]*)\] \"([^\"]*)\" (\d
++) ([\d\-]+) \"([^\"]*)\" \"([^\"]*)\" (\d+)}, [qw(h l u t r c b A R 
+T)] ],
102a103,104
>     } elsif ($optctl{debug} == 1) {
>       print STDERR $_;
[download]

"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Q312461; <HTML><A%
+20HREF="http://www.pghconnect.com/">www.pghconnect.com</a></HTML>)"
[download]

[reply]
[d/l]
[select]


Don't ask to ask, just ask
	PerlMonks