Apache and log4perl (high performance logging?)

zerohero has asked for the wisdom of the Perl Monks concerning the following question:

Could someone with experience provide some guidance as to the use of Log4perl w.r.t. Apache2 (and mod_perl, of course)? The basics are pretty obvious, and I know my way around the 3 technologies pretty well, but there are some performance related details I don't have experience with.

For example, I don't understand the performance impacts of (log4perl) logging in a high traffic web application. The idea of each of 32-64 Apache child processes contending for a lock (e.g. syswrite or semaphore via Log::Log4perl::Appender::Synchronized) and then writing to the disk seems to be an issue, I'm just not sure how big an issue it is. Should logging be minimal and then made verbose when one needs to interrogate? Or do people just run at full detail (who cares, no impact)? Some experience would provide illumination as to how important this is. Also knowing simple details like which appender to use based on experience, would be helpful (e.g. File + newsyslog, versus FileRotate).

Also interestingly, the informative Log4perl FAQ by messr. Mike Schilli mentions 30K ops/sec by an "in-memory" appender, which sounds high for something writing to disk. Call me a tad skeptical on this claim (perhaps it is a measurement of the library's efficiency sans disk writing). Mike also mentions newsyslog, an external program to do the log rolling, rather than using FileRotate, I'm guessing to parallelize and possibly do things like compression (again, I lack experience here, so this is just a guess).

I'd appreciate very much any responses from people with real world experience of what works well and what doesn't to help make the right choices for a scalable, maintainable, production system.

Comment on Apache and log4perl (high performance logging?)

Replies are listed 'Best First'.
Re: Apache and log4perl (high performance logging?) by andreas1234567 (Vicar) on Feb 11, 2009 at 09:20 UTC
I have used Log::Log4perl is production with Apache and mod_perl2, although not in very high traffic applications. So what are the performance impacts of (log4perl) logging in a high traffic web application? The answer is, as always, it depends. Running in `DEBUG` mode (full detail) can seriously slow down performance, so you should be careful when setting up your loggers, appenders and levels. Also, very verbose logging can make the logs hard to read, and you can keep less history than if you were less verbose. Benchmarking your application with different log4perl settings is important to understand how it affects you in your environment. Perhaps you should experiment with different disk setups, such as having `/var/log/` on a separate disk system? Your I/O performance will most likely be one of the most important factors. Do also keep an eye on the is_level functions in order to minimize impact of suppressed messages: `if($logger->is_error()) { $logger->error("Erroneous array: @super_long_array"); }` [download] -- No matter how great and destructive your problems may seem now, remember, you've probably only seen the tip of them. [1]	[reply] [d/l] [select]
Re: Apache and log4perl (high performance logging?) by jasonk (Parson) on Feb 11, 2009 at 20:53 UTC
Mike Schilli mentions 30K ops/sec by an "in-memory" appender, which sounds high for something writing to disk. Call me a tad skeptical on this claim (perhaps it is a measurement of the library's efficiency sans disk writing). An "in-memory" appender means exactly what it sounds like it means. It's not writing to disk, it's keeping the log entries in memory. If your performance concern is with the speed that log entries can be written to disk, there are plenty of alternatives to simple logfiles: Log::Log4perl::Appender::Spread - Log to a spread cluster Log::Log4perl::Appender::Jabber - Log to jabber Log::Log4perl::Appender::DBI - Log to a database Log::Log4perl::Appender::STOMP - Log to a message queue (such as POE::Component::MessageQueue) www.jasonkohles.com We're not surrounded, we're in a target-rich environment!	[reply]


The stupid question is the question not asked
	PerlMonks