Re: Question about the most efficient way to read Apache log files without All-In-One Modules from CPAN (personal learning exercise)

in reply to Question about the most efficient way to read Apache log files without All-In-One Modules from CPAN (personal learning exercise)

Apache log file formats (there are several) consist of space delimited variable width fields:

127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTT
+P/1.0" 200 2326 "http://www.example.com/start.html" "Mozilla/4.08 [en
+] (Win98; I ;Nav)"
[download]

(The last two quote strings may not be present.)

Every one of those fields except possibly the date and status code can vary in length; so pack and unpack which are primarily designed for fixed length fields and binary work, are the wrong tools for the job.

split is the more applicable tool for your task:

my @bits = split ' ', $line;
[download]

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority". I'm with torvalds on this

In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked

m with torvalds on this

Comment on Re: Question about the most efficient way to read Apache log files without All-In-One Modules from CPAN (personal learning exercise) Select or Download Code

Replies are listed 'Best First'.
Re^2: Question about the most efficient way to read Apache log files without All-In-One Modules from CPAN (personal learning exercise) by wrog (Friar) on Jun 17, 2015 at 04:58 UTC
`split` is fine as long as the character you're splitting on does not occur in your various fields. If it can, which then means there then has to be a scheme for quoting such fields or escaping such characters, then we're pretty much beyond what `split` can do and at the point where you need to be building the Regular Expression From Hell — which can be plenty fast if you do it right, but you have to do it right — or using Text::CSV or somesuch.	[reply] [d/l] [select]
Re^3: Question about the most efficient way to read Apache log files without All-In-One Modules from CPAN (personal learning exercise) by karlgoethebier (Abbot) on Jun 17, 2015 at 09:46 UTC
...we're pretty much beyond what split can do... Mh, we know the format: use Data::Dump; use feature qw(say); my $line =qq(127.0.0.1 - - [22/Apr/2015:13:35:04 +1000] "GET /bin/admi +n.pl HTTP/1.1" 401 509); my @bits = split /\s/, $line; dd\@bits; say qq(Host: $bits[0]); say qq(Logname: $bits[1]); say qq(User: $bits[2]); say qq(Time: $bits[3] $bits[4]); say qq(Request: $bits[5] $bits[6] $bits[7]); say qq(Status: $bits[8]); say qq(Size: $bits[9]); __END__ monks>apache.pl [ "127.0.0.1", "-", "-", "[22/Apr/2015:13:35:04", "+1000]", "\"GET", "/bin/admin.pl", "HTTP/1.1\"", 401, 509, ] Host: 127.0.0.1 Logname: - User: - Time: [22/Apr/2015:13:35:04 +1000] Request: "GET /bin/admin.pl HTTP/1.1" Status: 401 Size: 509 [download] Regards, Karl ŤThe Crux of the Biscuit is the Apostropheť	[reply] [d/l]
Re^4: Question about the most efficient way to read Apache log files without All-In-One Modules from CPAN (personal learning exercise) by lulz (Initiate) on Jun 17, 2015 at 19:13 UTC
Thanks for your reply! A quick question dealing with the internal workings of what you wrote: I understand that the split function can take any expression as its element then operate on the scalar, but what would be the more nuanced differences, particularly with memory usage and processing speed, if any, between using split and a general pattern match? Thanks!	[reply]
Re^5: Question about the most efficient way to read Apache log files without All-In-One Modules from CPAN (personal learning exercise) by karlgoethebier (Abbot) on Jun 17, 2015 at 19:53 UTC
Re^6: Question about the most efficient way to read Apache log files without All-In-One Modules from CPAN (personal learning exercise) by lulz (Initiate) on Jun 17, 2015 at 23:23 UTC
Re^4: Question about the most efficient way to read Apache log files without All-In-One Modules from CPAN (personal learning exercise) by wrog (Friar) on Jun 17, 2015 at 15:43 UTC
I'm not sure I'd want to bet my life that none of `logname`, `user` or the request URI can have spaces in them.	[reply] [d/l] [select]
Re^5: Question about the most efficient way to read Apache log files without All-In-One Modules from CPAN (personal learning exercise) by karlgoethebier (Abbot) on Jun 17, 2015 at 17:28 UTC
Re^6: Question about the most efficient way to read Apache log files without All-In-One Modules from CPAN (personal learning exercise) by wrog (Friar) on Jun 17, 2015 at 23:10 UTC
Re^3: Question about the most efficient way to read Apache log files without All-In-One Modules from CPAN (personal learning exercise) by BrowserUk (Patriarch) on Jun 17, 2015 at 10:08 UTC
He did ask for a learning exercise; not a pre-solved solution. Plus, chances are the he'll need to break the composite fields down further anyway, before he can do any analysis or storage. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked	[reply]
Re^4: Question about the most efficient way to read Apache log files without All-In-One Modules from CPAN (personal learning exercise) by karlgoethebier (Abbot) on Jun 17, 2015 at 17:21 UTC
"...He did ask for a learning exercise..." Yes, but i didn't reply to the OP. "...break the composite fields down..." Yes, sure. Perhaps like this: `karls-mac-mini:monks karl$ perl -E ' say split /[\[\]]/, qq([22/Apr/20 +15:13:35:04 +1000])' 22/Apr/2015:13:35:04 +1000 karls-mac-mini:monks karl$ perl -E ' say split /"/, qq("GET /bin/admin +.pl HTTP/1.1")' GET /bin/admin.pl HTTP/1.1 karls-mac-mini:monks karl$ perl -E 'say join "\t", split /\s/, qq(GET +/bin/admin.pl HTTP/1.1)' GET /bin/admin.pl HTTP/1.1 # usw...` [download] I just wanted to show wrog that a solution that only uses split is possible. Another question is ~~this it is desirable~~ if this is desirable. I guess some may call it abuse. Edit: Better wording. Best regards, Karl ŤThe Crux of the Biscuit is the Apostropheť	[reply] [d/l]
Re^5: Question about the most efficient way to read Apache log files without All-In-One Modules from CPAN (personal learning exercise) by BrowserUk (Patriarch) on Jun 17, 2015 at 18:10 UTC
Re^6: Question about the most efficient way to read Apache log files without All-In-One Modules from CPAN (personal learning exercise) by wrog (Friar) on Jun 17, 2015 at 23:27 UTC
Some notes below your chosen depth have not been shown here

In Section Seekers of Perl Wisdom