Re: Out of Memory Error -- Possible Leak?

How much memory do you have? Your data structures are immense. It looks like you have perhaps 100,000,000 arrays in the hash, and even if most are not there you are autovivifying their keys ( scalar(@array) different anonymous hashes) and a reference to an empty array.

All those structures cost fifty-some bytes each plus a hefty multiple of the number of data elements.

I don't see a leak, just prolific use of memory. Can you read and use less of the data at a time?

After Compline,
Zaxo

Comment on Re: Out of Memory Error -- Possible Leak?

Replies are listed 'Best First'.
Re^2: Out of Memory Error -- Possible Leak? by Anonymous Monk on Dec 13, 2005 at 23:56 UTC
The machine itself has 32 GB, but I'm using a 32-bit compiled Perl so presumably it can only address 4 of those. Also, 100 million arrays is a realistic estimate (its not sparse). I had thought that by using DBM::Deep I would be storing the majority of this data on disk, so that the actual amount stored in memory would be much smaller than 100 million x size(data-structure). Am I misguided on that?	[reply]
Re^2: Out of Memory Error -- Possible Leak? by emilper (Novice) on Dec 14, 2005 at 11:59 UTC
maybe it's because the garbage collector does not have time to run ...	[reply]
Re^3: Out of Memory Error -- Possible Leak? by robin (Chaplain) on Dec 15, 2005 at 12:37 UTC
That isn't possible, because Perl doesn't use a separate garbage collector. It simply reference-counts all its data (scalars, arrays, hashes etc) and frees an item when its reference count drops to zero. (This is why circular data structures cause memory leaks.)	[reply]
Re^4: Out of Memory Error -- Possible Leak? by emilper (Novice) on Dec 15, 2005 at 14:55 UTC
then how do you explain this ? try: perl -e ' use Tk; while (1) { my $top = new MainWindow; }' it will fill your RAM at a rate of about 1MB per second ...	[reply]
Re^5: Out of Memory Error -- Possible Leak? by robin (Chaplain) on Dec 16, 2005 at 01:31 UTC


The stupid question is the question not asked
	PerlMonks