-- Randal L. Schwartz, Perl hacker

HEHEHEHE, ok, I am going to take a few --'s for posting such a vapid post... but hot damn. I cracked up when I read this... that is SUCH a great way to sneak Perl into an organization.

Geek talking to manager...

"Don't worry, we won't be using any of those dirty open source tools here for our production system, we're just using them to prototype..."

6 weeks later...

"It turns out that Perl was 5 times faster and I was able to do it in 1/3rd the time... maybe we should extend your great vision in alowing me to use open source to prototype to our production environment, thereby showing a cost savings and helping our company!"

hehehehe... that's just eeeeeevil ++'s for merlyn

Plain and simple, for 99% of server log processing tasks that *I* have encountered, no matter what the size of the datafile, the bottleneck isn't going to be the processing of the data, it is going to be the IO caused by the reading and writing of said data, and that is in Perl, as it is in C++ a very optimized streamlined operation, only a few steps removed from a stdlib call.

Do the math, the time required for processing in memory of the lines that you read efficiently (i suggest looking into some of the block buffered reading hacks that we've discussed on this site this one was quite useful to us...) will greatly be outstripped by the time it simply takes to get the information in memory in the first place, reguarless of the language used to manipulate it (C, C++, or ASM aren't going to make your hard drive faster...) But having parsed data files approaching the terabyte size limit in *extremely* reasonable times, I have to tell you that I feel that Perl is more than qualified for 99% of these jobs.

Perl can be faster at reading than C using stdio. I haven't timed C++ using std:: I/O against C's stdio nor against Perl's I/O, so I can't say for sure, but I suspect that Perl can be faster than C++. Unfortunately, Perl for Linux is not properly optimized and so Perl's <STDIN> is about four-times slower than it should be.

Now, if the analysis required building data structures whose complexity was a function of the size of the log file, then I'd definitely have doubts about being able to do the analysis easily in Perl without running out of swap space (and running at a snail's pace before failing due to excessive page faulting). (Perl has a strong preference to consuming RAM in order to improve execution speed.) But that seems a very unlikely type of analysis.

So I'd have a very strong preference for doing this stuff in Perl unless you can dig up a more concrete reason why Perl isn't appropriate. A terrabyte log file in itself isn't a reason to not use Perl.

AgentM Systems nor Nasca Enterprises nor Bone::Easy nor Macperl is responsible for the comments made by AgentM. Remember, you can build any logical system with NOR.

--
$you = new YOU;
honk() if $you->love(perl)