http://qs321.pair.com?node_id=1130742

lulz has asked for the wisdom of the Perl Monks concerning the following question:

I have a question about the most efficient (in both memory and processing) way to read Apache log files. I realize that there are many many packages on CPAN that can read log files from Apache servers, but I am curious about what may be the best way to parse these files.

I am a beginner PERL programmer who is curious about the inner workings of both these packages and PERL itself.

I was thinking that it may be possible to use pack and unpack alongside File::Map somehow to be able to parse and process the log lines directly without loading the entire file into memory.

Is my line of thinking flawed? Is it possible to use pack and unpack in this manner with variable width files? I have read perlpacktut, but it only explains the capabilities of text files in a fixed width format.

Likewise, for the All-In-One packages for log parsing on CPAN, how do they parse these kinds of files and why?