Benchmark.pm ing -- and a little confused

Sifmole has asked for the wisdom of the Perl Monks concerning the following question:

The following block of cmpthese prints are confusing me a bit. If you look at the wallclock seconds there are very large ( 70% to 80% ) differences in time. However, the comparison percentages are 10% or less. Can someone explain to me what I am misunderstanding?


Benchmark: timing 100 iterations of smMod, smPlain...
     smMod: 198.103 wallclock secs (20.84 usr +  0.32 sys = 21.16 CPU)
+ @  4.72/s (n=100)
   smPlain: 1013.47 wallclock secs (22.88 usr +  0.39 sys = 23.27 CPU)
+ @  4.30/s (n=100)
          Rate smPlain   smMod
smPlain 4.30/s      --     -9%
smMod   4.72/s     10%      --
Benchmark: timing 100 iterations of seMod, sePlain...
     seMod: 161.968 wallclock secs (65.76 usr +  0.42 sys = 66.18 CPU)
+ @  1.51/s (n=100)
   sePlain: 435.528 wallclock secs (71.20 usr +  0.43 sys = 71.62 CPU)
+ @  1.40/s (n=100)
          Rate sePlain   seMod
sePlain 1.40/s      --     -8%
seMod   1.51/s      8%      --
Benchmark: timing 100 iterations of navMod, navPlain...
    navMod: 77.0209 wallclock secs (14.74 usr +  0.20 sys = 14.95 CPU)
+ @  6.69/s (n=100)
  navPlain: 317.475 wallclock secs (14.70 usr +  0.38 sys = 15.09 CPU)
+ @  6.63/s (n=100)
           Rate navPlain   navMod
navPlain 6.63/s       --      -1%
navMod   6.69/s       1%       --
[download]

As a note, in case anybody wonders or it is relevant, the different functions are requesting the same pages from two different servers. One is running mod_perl and the other is vanilla CGI. Thanks in advance

Comment on Benchmark.pm ing -- and a little confused Download Code

Replies are listed 'Best First'.
Re: Benchmark.pm ing -- and a little confused by Roy Johnson (Monsignor) on Mar 14, 2005 at 21:49 UTC
It's comparing CPU time, which is a better measure of how long it takes your program to run than ordinary time. Caution: Contents may have been coded under pressure.	[reply]
Re^2: Benchmark.pm ing -- and a little confused by Sifmole (Chaplain) on Mar 14, 2005 at 21:57 UTC
So I guess, because a large portion of the actual processing time is on a remote server and the local server is just waiting for most of it's "wallclock" time that is the reason for the large difference in percentages? So in this case the wallclock time would actually be a better gauge of the comparison than the CPU time, right?	[reply]
Re^3: Benchmark.pm ing -- and a little confused by Roy Johnson (Monsignor) on Mar 14, 2005 at 22:02 UTC
Yes, if the code of interest is being run on a remote machine, then wallclock seconds are your best measure. Caution: Contents may have been coded under pressure.	[reply]
Re^4: Benchmark.pm ing -- and a little confused by halley (Prior) on Mar 15, 2005 at 17:40 UTC
Re: Benchmark.pm ing -- and a little confused by Joost (Canon) on Mar 14, 2005 at 21:58 UTC
If you are requestion pages from machines other than the tests are run on, the user/sys times are not relevant (if the requests are done the same way). The user/sys time is the amount of CPU time the tests take to run. If a process is waiting for something in a syscall (i.e. waiting for network input) the scheduler usually switches to anytother process, so the actual CPU time for that process will not increase. The wallclock will increase, since that is the actual amount of physical time that is spend (the clock on the wall). In other words, if you want to measure the responsiveness of your servers, your current setup is good, and you can more or less disregard the CPU time on your test machine and just watch the wallclock time. You should make sure you get no interference from other processes on your test machine and servers, though. One of the advantages of measuring in CPU time is that if your machine is really busy running other processes, the CPU time for a test will not be influenced by it. But it doesn't really work "cross-process" and "cross-network". "What should it profit a man, if he should win a flame war, yet lose his cool?"	[reply]


No such thing as a small change
	PerlMonks