You can't after the fact, but you can trigger dropping into the debugger from code if you've started the program under it to begin with.

• Make a signal handler for, say, SIGUSR1 which sets $DB::single = 1;
• Start with perl -d mydaemon regular arguments and just hit "c" when you get the debugger prompt
• When you want the program to drop into the debugger, send it your signal (kill -USR1 pid) and you should get dropped into the debugger inside your signal handler.
• Profit! Debug!

What OS are you running on? If Linux, I would first try strace (hey, maybe you'll get lucky :)). If you can't predict when the problem will occur, pipe the strace output to a perl script that rolls the file when it gets big, and keeps the last 10 or so around. If you are running on Solaris, you will probably have more success with dtrace, but this is non-trivial as it has its own programming language ('D').

Looks like the problem is in user space. Most probably a worst-case scenario (combinatorial explosion) of a regexp is triggered. So strace won't help. And this is FreeBSD.
--kap

Try this. Save it as yourperl/site/lib/Devel/Trace/Remote.pm

Updated: Improved the cleanup to allow reconnects.

ppackage Devel::Trace::Remote;
use strict;
use warnings;
use IO::Socket;
use threads;
use threads::shared;
use Thread::Queue;

my $connected :shared = 0;

my $Q = new Thread::Queue;

sub DB::DB {
    return unless $connected;
    $Q->enqueue( join ' : ', caller );
    return;
}

async {
    my $server = IO::Socket::INET->new(
        LocalHost => 'localhost:54321',
        Listen => 1,
        Reuse => 1,
    ) or die $!;
    $server->autoflush;
    while( my $client = $server->accept ) {
        $connected = 1;
        while( $_ = $Q->dequeue ) {
            print $client "$_\n\r" or last;
        };
        $connected = 0;
        $Q->dequeue while $Q->pending;
    }
}->detach;

1;
[download]

Then start your script using perl -d:Trace::Remote yourscript. When things start going awry, telnet into localhost:54321 and you will get output along the lines of:

##pkg  file      lineno
main : ack1.pl : 21
main : ack1.pl : 22
main : ack1.pl : 21
main : ack1.pl : 22
main : ack1.pl : 21
main : ack1.pl : 22
main : ack1.pl : 21
main : ack1.pl : 22
main : ack1.pl : 21
main : ack1.pl : 22
main : ack1.pl : 21
main : ack1.pl : 22
main : ack1.pl : 21
main : ack1.pl : 22
main : ack1.pl : 21
main : ack1.pl : 22
main : ack1.pl : 21
[download]

With nothing connected, it will have minimal impact on the process. (For cpu-bound processes about 4 1/2 times slower. Much less for a IO-bound process. When you are connected, expect the slowdown to roughly double.) You may also see some memory growth if the program is is a tight loop (generating output faster than telnet can receive it.

Many enhancements are possible, but this should allow you to get a feel for what is going on inside the code without to much impact on the normal running. It may end up on CPAN if you find it useful.

I mean you can make test version of your script and watch what happens with it.

Alas, Devel::Profile is too slow to run under in production and the problem is only seen under heavy production load :(
--kap

OK. I think it should be possible to test the script on non-production machine. If so, then you can search bottleneck by commenting out different pieces of code until the heavy load cases disappear.

Pay more attention to regexps that can be bad written like this

('ab' x 3000 . 'c') =~ /.*ab.*c/; # fast if can match
('ab' x 3000)       =~ /.*ab.*c/; # very very very slow if no match
[download]

... the problem is only seen under heavy production load

Have you checked the memory consumption (either on a single instance of the script, or summed over multiple instances that might be running simultaneously on a given server)?

If any single instance takes up a significant amount of memory, then you can probably work out how many concurrent jobs it would take to swamp available RAM, and cause one or more of the jobs to go into severe swapping / page faulting. The standard unix/gnu "top" command might suffice to spot a problem of that sort as it's happening.

How hard will it be to reduce the memory footprint of your script? Alternately, how bad will it be to limit the number of simultaneous jobs? The thing about page-fault delays is that the timing impact is non-linear: making one job wait a few sec before it can really start -- to make sure that it is serialized relative to some other job (won't start till some other job finishes) -- can often lead to an overall faster completion than allowing it to run immediately and simultaneously, causing unsustainable competition for available resources,

In addition to the other good advice already provided, you may want to look at using Devel::NYTProf, an interesting screencast on profiling and a demo of this module in use can be found here.

Hope this helps

Martin

I'm a big fan of Log::Log4perl .. it's logging you can turn on and off (or rather, up or down in sensitivity) externally from the application that's running.