Re: Optimizing a large project.

First, is the problem that it takes too long, or that it uses too much CPU? If it's that it takes too long, you need to profile wall time, not CPU time. If your application does any significant I/O, that's almost certainly the time bottleneck, and you won't see it when profiling CPU time. Until you've done this, forget about method call overhead and inline functions.

Next, stop using fields. The feature you're referring to was called pseudohashes and it was killed a long time ago (5.8 I think) because it caused problems and didn't deliver on the expected improvements. If you want hashes that don't autovivify keys, use Hash::Util::lock_keys(). If you're not on 5.8 yet, it would be a good move to get there.

Comment on Re: Optimizing a large project.

Replies are listed 'Best First'.
Re^2: Optimizing a large project. by dextius (Monk) on Jun 12, 2008 at 19:09 UTC
Our processes do not run and shut down. They start up in the morning, process all events given to it, and then shut down in the afternoon. We're measuring the response time between the timestamp of the event and when we responded to it. We do have fairly significant IO, both to disk and the network. The files are buffered writes, so we don't think that's our problem. Wow. Fields is that bad huh? I'll look into lock_keys. I need to re-read the 5.9 fields note on the perldoc, I think they mentioned the moved away from pseudohashes..	[reply]
Re^3: Optimizing a large project. by perrin (Chancellor) on Jun 12, 2008 at 19:25 UTC
We're measuring the response time between the timestamp of the event and when we responded to it. So, that would be wall time. Profile with wall time then. We do have fairly significant IO, both to disk and the network. The files are buffered writes, so we don't think that's our problem. If you never measured it, you don't really know. I/O is nearly always the problem on modern hardware, unless you're doing 3D rendering or something similarly compute intensive.	[reply]
Re^4: Optimizing a large project. by dextius (Monk) on Jun 12, 2008 at 21:00 UTC
I'll tackle the I/O benchmarks immediately, do you have a suggestion on a specific approach by chance?	[reply]
Re^5: Optimizing a large project. by perrin (Chancellor) on Jun 12, 2008 at 22:05 UTC
Re^3: Optimizing a large project. by salva (Canon) on Jun 13, 2008 at 08:15 UTC
To know where the problem lies, if in IO or in CPU usage, all you need to do is to run `top` or any other similar monitoring tool while your program is running. Roughly, if it shows that CPU usage goes over 90%, then your problem is CPU, below 30% it is IO and in the middle it is both. Well, this method will not work if your application just receives a few events per minute that are processed in less than a second, below the `top` (and your eyes!) resolution.	[reply] [d/l] [select]
Re^4: Optimizing a large project. by salva (Canon) on Jun 13, 2008 at 08:48 UTC
In the later case, to calculate the CPU usage ratio, you can use something like that: `my ($cpu_time, $clock_time) = (0, 0); sub your_event_processing { my ($start_user, $start_system) = times; my $start_clock = time; # do the event processing here # ... my ($end_user,$end_system) = times; my $end_clock = time; $clock_time += ($end_clock - $start_clock); $cpu_time += ($end_user - $start_user + end_system - $start_system); } END { printf STDERR "clock_time: %d, cpu_time: %d (%4.2f%%)\n", $clock_time, $cpu_time, $cpu_time * 100 / $clock_time; }` [download] And let it run for some hundreds of events, so that the rounding errors go away.	[reply] [d/l]


Pathologically Eclectic Rubbish Lister
	PerlMonks