We're measuring the response time between the timestamp of the event and when we responded to it.

We do have fairly significant IO, both to disk and the network. The files are buffered writes, so we don't think that's our problem.

I'll tackle the I/O benchmarks immediately, do you have a suggestion on a specific approach by chance?

I don't know how the NYT profiler does things, but with Devel::DProf all you have to do is take the data you already have and feed it to dprofpp -r. The -r flag means "real" time.

UPDATE: Looks like NYTProf makes you declare this before you collect the profile data. The option is called "useclocktime".

Well, this method will not work if your application just receives a few events per minute that are processed in less than a second, below the top (and your eyes!) resolution.

In the later case, to calculate the CPU usage ratio, you can use something like that:

my ($cpu_time, $clock_time) = (0, 0);

sub your_event_processing {
  my ($start_user, $start_system) = times;
  my $start_clock = time;

  # do the event processing here
  # ...

  my ($end_user,$end_system) = times;
  my $end_clock = time;

  $clock_time += ($end_clock - $start_clock);
  $cpu_time += ($end_user - $start_user + end_system - $start_system);
}

END {
  printf STDERR "clock_time: %d, cpu_time: %d (%4.2f%%)\n",
                $clock_time, $cpu_time, $cpu_time * 100 / $clock_time;
}
[download]

And let it run for some hundreds of events, so that the rounding errors go away.