http://qs321.pair.com?node_id=1000797


in reply to Processing ~1 Trillion records

I second the advice of benchmarking/profiling your existing code, find existing bottlenecks. Perhaps the script in question is inefficient, profiling will reveal what's going on. Devel::NYTProf/Debugging and Optimization.