Re: Memory efficient way to deal with really large arrays?

Instead of using an array of arrays for storing the pairs, you can use two arrays, one for the first elements and another for the second ones. That is going to reduce your memory consumption by an order of magnitude:

my @d;
my (@p1, @p2);
for (my $i=0;$i<500000000;$i++)
{
   my ($p1,$p2) = getPair();
   $d[$p1]++;
   $d[$p2]++;
   push @p1, $p1;
   push @p2, $p2;
}
[download]

Comment on Re: Memory efficient way to deal with really large arrays? Download Code

Replies are listed 'Best First'.
Re^2: Memory efficient way to deal with really large arrays? by LanX (Saint) on Dec 13, 2020 at 13:01 UTC
That's brilliant. It's simple and very efficient. Another approach would have been to `pack` the pairs into a 64 bit string to be stored in one array, that might be more efficient but wouldn't be as simple as this trick. Cheers Rolf _{(addicted to the Perl Programming Language :) Wikisyntax for the Monastery}	[reply]
Re^3: Memory efficient way to deal with really large arrays? by salva (Canon) on Dec 13, 2020 at 13:21 UTC
that might be more efficient I am not sure. Some years ago Nicholas Clark introduced a change in the internal representation of SVs that made then very efficient for integers and doubles. Briefly, all internal types used to have a fixed size head (SV_HEAD) and a variable sized body. That change, used a trick to embed the body inside the header for SVs representing numbers, making then more cache friendly and reducing memory usage by eliminating the body part. So, having two arrays with numbers may actually use less memory than an array of small strings.	[reply]


Pathologically Eclectic Rubbish Lister
	PerlMonks