Caching hash

vit has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks
I use ActiveState perl and want to cache a big hash in order not to read it from the file every time when I execute my perl program.
I also need to be able to pick it up on the subsequent calls of CGI on the server side.
What could you suggest?

I tried

IPC::SharedCache
IPC::SharedMemory
IPC::Shareable
[download]

but first I am not sure if this is what I need and second failed to install these modules.

Comment on Caching hash Download Code

Replies are listed 'Best First'.
Re: Caching hash by moritz (Cardinal) on Jun 29, 2009 at 20:35 UTC
You can also look at the various memcached libraries, which even allows you to keep the cache on a different machine, or many, if necessary. I've heard good thing about it, but I haven't used it yet myself.	[reply]
Re^2: Caching hash by jeanluca (Deacon) on Jun 29, 2009 at 20:57 UTC
I've used memcached and it is really great! However, I would only use it if a DB is not an option! Cheers LuCa	[reply]
Re: Caching hash by gaal (Parson) on Jun 29, 2009 at 20:35 UTC
What's being slow, the IO of reading it from disk, or do you have some calculation going on in making it? How is it actually stored on disk? The fastest format for that is probably `Storable::nstore(\%data, $file); ... Storable::retrieve($file)`. If you have some long-running process that can hold a copy of the data and you want to use that, is a real copy (over a named pipe or something) really too expensive? How big is your data? Is it immutable?	[reply] [d/l]
Re^2: Caching hash by vit (Friar) on Jun 29, 2009 at 20:45 UTC
The reading is slow. I use Storable and the data size is around 1g. It is processed (calculations) in less then one sec, but read much longer.	[reply]
Re^3: Caching hash by gaal (Parson) on Jun 30, 2009 at 05:22 UTC
That's a lot of data! I bet you don't need it all at once on the transient CGIs, though. Should your long-running process provide a query interface? That way the random scripts that come and go only ask it for the stuff they need. You can memcache the results of these queries if you like. No need to spend 1gb on the cache: experiment with some values, see what your hit rates are.	[reply]
Re: Caching hash by Arunbear (Prior) on Jun 29, 2009 at 20:38 UTC
Did you try DB_File ?	[reply]
Re^2: Caching hash by perrin (Chancellor) on Jun 29, 2009 at 20:49 UTC
Seconded. Or, more precisely, BerkeleyDB.	[reply]
Re^2: Caching hash by vit (Friar) on Jun 29, 2009 at 20:51 UTC
This is associated with BerkleyDB. I did not see there anything about keeping the DB cached between sessions or shearing between processes, or it is possible?	[reply]
Re^3: Caching hash by Arunbear (Prior) on Jun 29, 2009 at 21:21 UTC
The DB would be stored on the filesystem, so you can access it via its name between sessions or processes. It would be up to you to manage 'expiry' of the data, unless you use a wrapper like Cache::BerkeleyDB	[reply]
Re^4: Caching hash by vit (Friar) on Jun 29, 2009 at 21:48 UTC
Re^5: Caching hash by Arunbear (Prior) on Jun 29, 2009 at 22:08 UTC
Re^5: Caching hash by Marshall (Canon) on Jun 30, 2009 at 06:52 UTC

Back to Seekers of Perl Wisdom