What's wrong with Storable

Hi,

I can only wonder about the memory and CPU requirements...
Saving a bigger hash (ca. 515000 entries, where each key is about 7-10 chars long and each value is a Tree::Nary structure), results in a ca. 330MB file. Uncompressed that is.

Restoring it, takes an average of 3,8 times more memory than disk space => ca. 1,2GB!

Saving is weird, after all the 330MB have been saved, store takes another 2200 seconds of 100% CPU time on a P3 1,26GHz even after the complete file is already on disk.

Moreover, store seems to take when storing the same amount of memory as the data structure takes. in the aforementioned case over 2,1GB RAM are allocated! Gosh...

Bye
PetaMem All Perl: MT, NLP, NLU

Comment on What's wrong with Storable

Replies are listed 'Best First'.
Re: What's wrong with Storable by eserte (Deacon) on Apr 16, 2004 at 23:50 UTC
Restoring it, takes an average of 3,8 times more memory than disk space => ca. 1,2GB! This is not surprising --- Storable may store the data more space-efficient than it is in the RAM --- think of malloc overhead, necessary struct alignment, and the size of a SV. Saving is weird, after all the 330MB have been saved, store takes another 2200 seconds of 100% CPU time on a P3 1,26GHz even after the complete file is already on disk.* Maybe it's the global destruction which is causing the slowness. This may be due to an inefficient malloc. For example, it was reported that on FreeBSD, perl with the system malloc is very slow on deleting a large hash. Similar is true for certain Linux' malloc versions.	[reply]
Re: Re: What's wrong with Storable by PetaMem (Priest) on Jun 01, 2004 at 14:42 UTC
Actually it is what I would call a design flaw of the Storable module, this was discussed (and resolved) in detail at the perl5 porters list. Expect a patch to storable RSN. Bye PetaMem All Perl: MT, NLP, NLU	[reply]


Welcome to the Monastery
	PerlMonks