Finding memory usage of perl structure efficiently

I always wanted to know memory usage of particular structure in my perl programs. Recently, I noticed that Storable file size corresponds nicely to memory usage of my application, so I used that as a hint of memory usage before loading data.

Then, I decided to show usage of already loaded data. Using Devel::Size total_size seemed like the way to go, but it's much slower than simple Storable piping to /dev/null

             Rate total_size   storable
total_size 4.04/s         --       -70%
storable   13.5/s       234%         --
[download]

using this code

use Benchmark qw(:all);

use Devel::Size;
use Storable;

sub _storable_size {
        open(my $fh, '|-', 'cat > /dev/null');
        Storable::store_fd $_[0], $fh;
        tell($fh);
}

my $o;
$o->{$_} = rand() foreach ( 'a' .. 'z' );
my $hash;
$hash->{$_} = $o foreach ( 0 .. 100_000 );

cmpthese( 100, {
        'total_size' => sub { Devel::Size::total_size( $hash ) },
        'storable'   => sub { _storable_size( $hash ) },
});
[download]

To be honest, I didn't expect so much difference. I can probably setup pipes myself to get a bit more performance, but I would really like to know alternative ways to find out perl memory usage without performance impact.

I'm experimenting with this code in gist on github as a log of things that I tried.

2share!2flame...

Comment on Finding memory usage of perl structure efficiently Select or Download Code

Replies are listed 'Best First'.
Re: Finding memory usage of perl structure efficiently by BrowserUk (Patriarch) on Jun 26, 2010 at 19:44 UTC
Just a thought. Have you tried Devel::Size v 0.72? You should find it uses far less memory for it own purposes, which means it runs much more quickly. It also gives more accurate results and is far less likely to cause an exception than the previous versions. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. RIP an inspiration; A true Folk's Guy	[reply]
Re^2: Finding memory usage of perl structure efficiently by Burak (Chaplain) on Jun 27, 2010 at 00:32 UTC
It's not easy to find as it's marked as "unauthorized". search.cpan.org or cpan shell search does not show it. You need to specifically search for it to find. So, why not re-release this as an official 0.73 as Tels already marked this distro as abandonware^{1}? Apart from improvements, I know that you also fixed some serious bugs in it.	[reply]
Re^3: Finding memory usage of perl structure efficiently by BrowserUk (Patriarch) on Jun 27, 2010 at 01:00 UTC
It's not easy to find as it's marked as "unauthorized". search.cpan.org or cpan shell search does not show it. When I look at Devel::Size, 0.72 is linked as the "latest release". So, why not re-release this as an official 0.73 as Tels already marked this distro as abandonware{1}? I'm not sure how that would be any different from having already released it as version 0.72? I did offer to take it over, but apparently the existing two co-maintainers couldn't agree to that. Apart from improvements, I know that you also fixed some serious bugs in it. At the point where I released what I had after failing to get it adopted as "official" (whatever that means), I believe I had fixed every outstanding bug on RT, including several that had been around for years; I added exception handling to avoid the frequent traps; and made it 64-bit compilable. Dunno what more I could have done? Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. RIP an inspiration; A true Folk's Guy	[reply]
Re^4: Finding memory usage of perl structure efficiently by Burak (Chaplain) on Jun 27, 2010 at 01:39 UTC
Re^2: Finding memory usage of perl structure efficiently by dpavlin (Friar) on Jun 27, 2010 at 13:20 UTC
Thanks for lot for a pointer to newer version. On first run, it reported `Devel::Size: Please rebuild D::S with TRACKING_SLOTS > 15814` [download] Fiar enough, but overall it's fastest method to find size of perl structure: # 20000 elements Rate size_0.71 reopen cat size_0.72 freeze size_0.71 80.3/s -- -43% -65% -83% -84% reopen 142/s 77% -- -38% -71% -71% cat 227/s 183% 60% -- -53% -54% size_0.72 485/s 505% 242% 114% -- -1% freeze 490/s 511% 245% 116% 1% -- # 30000 elements Rate size_0.71 reopen cat freeze size_0.72 size_0.71 37.1/s -- -54% -65% -83% -84% reopen 80.4/s 117% -- -25% -63% -66% cat 107/s 189% 33% -- -51% -54% freeze 217/s 486% 170% 103% -- -7% size_0.72 234/s 530% 191% 118% 7% -- # 40000 elements Rate size_0.71 reopen cat freeze size_0.72 size_0.71 22.1/s -- -59% -67% -83% -85% reopen 54.1/s 144% -- -20% -58% -64% cat 67.4/s 204% 25% -- -48% -55% freeze 130/s 487% 141% 93% -- -13% size_0.72 149/s 571% 175% 121% 14% -- # 50000 elements Rate size_0.71 reopen cat freeze size_0.72 size_0.71 13.6/s -- -63% -70% -81% -85% reopen 36.5/s 169% -- -20% -48% -61% cat 45.8/s 237% 25% -- -35% -51% freeze 70.2/s 417% 92% 53% -- -24% size_0.72 92.6/s 582% 153% 102% 32% -- # 60000 elements Rate size_0.71 reopen cat freeze size_0.72 size_0.71 9.97/s -- -66% -69% -80% -86% reopen 28.9/s 190% -- -10% -43% -59% cat 31.9/s 220% 11% -- -37% -55% freeze 50.5/s 407% 75% 58% -- -28% size_0.72 70.4/s 606% 144% 120% 39% -- # 70000 elements Rate size_0.71 reopen cat freeze size_0.72 size_0.71 7.91/s -- -65% -69% -79% -86% reopen 22.7/s 187% -- -12% -41% -59% cat 25.9/s 227% 14% -- -32% -54% freeze 38.2/s 383% 68% 48% -- -32% size_0.72 56.1/s 609% 147% 117% 47% -- # 80000 elements Rate size_0.71 reopen cat freeze size_0.72 size_0.71 5.78/s -- -69% -72% -80% -86% reopen 18.9/s 227% -- -10% -35% -55% cat 21.0/s 264% 11% -- -28% -50% freeze 29.1/s 404% 54% 39% -- -31% size_0.72 42.3/s 632% 124% 101% 45% -- # 90000 elements Rate size_0.71 reopen cat freeze size_0.72 size_0.71 4.95/s -- -69% -72% -80% -86% reopen 16.1/s 226% -- -9% -35% -55% cat 17.7/s 257% 9% -- -29% -51% freeze 24.7/s 399% 53% 40% -- -32% size_0.72 36.1/s 630% 124% 105% 46% -- # 100000 elements Rate size_0.71 reopen cat freeze size_0.72 size_0.71 4.21/s -- -69% -72% -80% -86% reopen 13.8/s 227% -- -9% -35% -56% cat 15.2/s 261% 10% -- -29% -51% freeze 21.3/s 407% 55% 40% -- -31% size_0.72 31.0/s 637% 125% 104% 45% -- [download] I would love to see version 0.71 depreciated :-) 2share!2flame...	[reply] [d/l] [select]
Re^3: Finding memory usage of perl structure efficiently by BrowserUk (Patriarch) on Jun 27, 2010 at 15:30 UTC
On first run, it reported `Devel::Size: Please rebuild D::S with TRACKING_SLOTS > 15814` I assume that you are running a 64-bit platform with 8GB of ram? My version requires a small, static area of memory for tracking purposes. This has to be sized to cater for the size of the memory of the machine. It ships with this sized to cater for the maximum memory on a 32-bit machine (8192 * sizeof(void *). On a 64-bit machine that has more than 4GB, it requires this be increased. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. RIP an inspiration; A true Folk's Guy	[reply] [d/l]
Re^4: Finding memory usage of perl structure efficiently by dpavlin (Friar) on Jun 27, 2010 at 22:20 UTC
Re^5: Finding memory usage of perl structure efficiently by BrowserUk (Patriarch) on Jun 28, 2010 at 04:47 UTC
Re: Finding memory usage of perl structure efficiently by zwon (Abbot) on Jun 26, 2010 at 19:07 UTC
You can rewrite `_storable_size` as follows: `sub _storable_size { length Storable::freeze $_[0]; }` [download]	[reply] [d/l] [select]
Re^2: Finding memory usage of perl structure efficiently by dpavlin (Friar) on Jun 27, 2010 at 11:14 UTC
This is simplier, but it would take additional memory to freeze my structure (which is ~120Mb or so) 2share!2flame...	[reply]

Back to Meditations