I've been building web applications for quite a few years now. One of the things I always feel lacking is the live web application performance monitoring. We spend lots of time (here and other sites) learning about various coding/design principles that help performance. But a live application behave quite differently from subroutine benchmarks/profiling. Often, when someone (e.g. a customer) alerts me of a inconsistent performance problem, I'm frankly at a loss as to (1) what is causing the problem, and (2) how to resolve it. Since such problems go away either after restart or under lower load, it's pretty difficult to look through code to catch it.
I'm wondering what you monks do for your live applications? What do you really do --as opposed to "I'll take a look"-- when someone tells you "hey, your app is really slow now"? Are there tools available? Can DProf being run in CGI/mod_perl and with how much overhead? Of couse you can log some timing for, say db calls, but it's too intrusive (requires littling the code with such logging.)
I've read some tools like those from Wily Technology for java application monitoring, I think they do some bytecode instrumentation to accomplish that. Does or can perl have such tools? There is a APM (Application Portfolio Monitoring) industry, does it serve the perl market?