You mentioned that you store most of the data in hashes. Why not try only loading data into the hash when it is asked for, using the hash as a cache? Then you get the speed of the hash without loading all of the data into the hash, when you may only need part of it.
An example cribbed more or less from page 609 of Programming Perl, 3d edition:
#!/usr/bin/perl -w
use strict;
use warnings;
sub get_data {
# here we simulate the expensive operation
# of loading data into the hash
my $idx = shift;
print "retrieving data for $idx...\n";
return "value for $idx";
}
my $result;
my %cache;
foreach (1,2,4,16,32,2,64,1,2) {
# here we are asking for values from the cache
# (hash), and if they aren't there, we get them
# from the file.
$result = $cache{$_} ||= get_data($_);
print "$result\n";
}
The output from this program is:
retrieving data for 1...
value for 1
retrieving data for 2...
value for 2
retrieving data for 4...
value for 4
retrieving data for 16...
value for 16
retrieving data for 32...
value for 32
value for 2
retrieving data for 64...
value for 64
value for 1
value for 2
As you can see, after the first time we ask for an element, we no longer have to go back to the file to get the data. Neat, eh? I hope that helps you. -
zeno