Make Perl use real malloc

creamygoodness has asked for the wisdom of the Perl Monks concerning the following question:

Greets,

To catch memory leaks in the Perl bindings for Apache Lucy, we run the entire test suite under Valgrind. It takes a long time to complete, but it's an effective technique for catching problems in our C code.

However, Valgrind doesn't ordinarily catch leaks of Perl/XS data structures. As I understand things, Valgrind overrides malloc/calloc/realloc/free and installs its own versions (which track memory usage). However, if you have your own allocation scheme, as Perl does, Valgrind may not notice when things go awry.

Say that your custom allocator requests blocks in large chunks from the OS, and then frees them during a cleanup phase just prior to process exit. Valgrind will likely notice if you fail to free one of those large chunks -- but it won't notice if memory parceled out by your allocator leaks during ordinary operation. I believe that describes the situation with Perl.

The Test::LeakTrace module from CPAN has caught memory leaks for us that Valgrind has not -- for instance, a qr// structure created from Perl-space that our XS binding code leaked a refcount for. Theoretically, we might consider running our test suite under Test::LeakTrace, just as we run it under Valgrind, in order to perform a thorough check. However, what I would prefer to do is compile a custom debugging Perl without the custom allocator, so that allocation of Perl data structures goes through malloc/free/etc. and our existing usage of Valgrind can catch these leaks.

Looking at the file malloc.c in the Perl source (I'm using the tarball for 5.12.2), it seems that definining the NO_FANCY_MALLOC and PLAIN_MALLOC symbols might achieve this goal.

  There are two macros which serve as bulk disablers of advanced
  features of this malloc: NO_FANCY_MALLOC, PLAIN_MALLOC (undef by
  default).  Look in the list of default values below to understand
  their exact effect.  Defining NO_FANCY_MALLOC returns malloc.c to th
+e
  state of the malloc in Perl 5.004.  Additionally defining PLAIN_MALL
+OC
  returns it to the state as of Perl 5.000.
[download]

I tried running ./Configure with those options:

./Configure -DDEBUGGING -Dprefix=/usr/local/debugperl \
-DNO_FANCY_MALLOC -DPLAIN_MALLOC -es
[download]

It doesn't seem to have worked, so maybe I have to specify those symbols using something other than the command line.

Before I go try again, does our analysis seem sound and our approach seem promising? I haven't yet grokked the details of Perl's deep dark magic memory allocation.

Comment on Make Perl use real malloc Select or Download Code

Replies are listed 'Best First'.
Re: Make Perl use real malloc by ikegami (Patriarch) on Mar 15, 2011 at 05:58 UTC
To pass defines the the C compiler rather than the configuration system, use `sh Configure -Accflags="-D... -D..."` [download] Note that one of Configure's prompts asks what allocator you want to use. Wouldn't that be a better way to go? But I thought valgrind worked with the default configuration on linux? Good luck!	[reply] [d/l]
Re^2: Make Perl use real malloc by creamygoodness (Curate) on Mar 15, 2011 at 16:16 UTC
It used to be that the FreeBSD port of Valgrind couldn't work with the system Perl, but that was because the FreeBSD system Perl was built with `-Dusemymalloc`, which Valgrind can't handle. I don't know of a similar problem with any system Perl in a Linux distro, nor have I seen any problems with a system Perl on OS X. Valgrind works with Perl on all those systems. It also works with Perl on FreeBSD if you custom compile your own Perl without `-Dusemymalloc`. The only Configure prompts I know of relating to malloc() control `PERL_MALLOC_WRAP` and `usemymalloc`: `Do you wish to wrap malloc calls to protect against potential overflow +s? [y] n Do you wish to attempt to use the malloc that comes with perl5? [n]` [download] If we want to work with Valgrind, we definitely don't want to enable `usemymalloc` (which is disabled by default). The `Accflags="-DNO_FANCY_MALLOC -DPLAIN_MALLOC "` Configure flag worked as intended -- thanks for the tip! Unfortunately, it did not produce a Perl that allowed us to use Valgrind in place of Test::LeakTrace. Maybe what we want is impossible because of how the global destruction phase works in Perl. If we use a debugging Perl and set `PERL_DESTRUCT_LEVEL` to 2, Perl will clean up all scalars, regardless of their refcounts. If we don't set `PERL_DESTRUCT_LEVEL`, it drops everything on the floor and lets the process `exit()` clean things up. There's no middle ground where it leaves only scalars with leaked refcounts behind.	[reply] [d/l] [select]
Re^3: Make Perl use real malloc by ikegami (Patriarch) on Mar 15, 2011 at 17:11 UTC
No matter what allocator Perl uses, valgrind can't detect leaked SVs (e.g. reference cycles, refcount to high) if Perl is always aware of its SVs, and that has to be the case since Perl attempts to free them during global destruction. $ valgrind perl -e'my $x; $x = \$x; undef $x if $ARGV[0]' 1 ==21363== Memcheck, a memory error detector ==21363== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et +al. ==21363== Using Valgrind-3.6.0.SVN-Debian and LibVEX; rerun with -h fo +r copyright info ==21363== Command: perl -emy\ $x;\ $x\ =\ \\$x;\ undef\ $x\ if\ $ARGV[ +0] 1 ==21363== ==21363== Warning: bad signal number 0 in sigaction() ==21363== ==21363== HEAP SUMMARY: ==21363== in use at exit: 80,794 bytes in 609 blocks ==21363== total heap usage: 786 allocs, 177 frees, 93,378 bytes allo +cated ==21363== ==21363== LEAK SUMMARY: ==21363== definitely lost: 0 bytes in 0 blocks ==21363== indirectly lost: 0 bytes in 0 blocks ==21363== possibly lost: 14,538 bytes in 354 blocks ==21363== still reachable: 66,256 bytes in 255 blocks ==21363== suppressed: 0 bytes in 0 blocks ==21363== Rerun with --leak-check=full to see details of leaked memory ==21363== ==21363== For counts of detected and suppressed errors, rerun with: -v ==21363== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 22 from + 7) $ valgrind perl -e'my $x; $x = \$x; undef $x if $ARGV[0]' 0 ==21366== Memcheck, a memory error detector ==21366== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et +al. ==21366== Using Valgrind-3.6.0.SVN-Debian and LibVEX; rerun with -h fo +r copyright info ==21366== Command: perl -emy\ $x;\ $x\ =\ \\$x;\ undef\ $x\ if\ $ARGV[ +0] 0 ==21366== ==21366== Warning: bad signal number 0 in sigaction() ==21366== ==21366== HEAP SUMMARY: ==21366== in use at exit: 80,794 bytes in 609 blocks ==21366== total heap usage: 786 allocs, 177 frees, 93,378 bytes allo +cated ==21366== ==21366== LEAK SUMMARY: ==21366== definitely lost: 0 bytes in 0 blocks ==21366== indirectly lost: 0 bytes in 0 blocks ==21366== possibly lost: 14,538 bytes in 354 blocks ==21366== still reachable: 66,256 bytes in 255 blocks ==21366== suppressed: 0 bytes in 0 blocks ==21366== Rerun with --leak-check=full to see details of leaked memory ==21366== ==21366== For counts of detected and suppressed errors, rerun with: -v ==21366== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 22 from + 7) [download]	[reply] [d/l]
Re^4: Make Perl use real malloc by creamygoodness (Curate) on Mar 15, 2011 at 18:56 UTC
Re^5: Make Perl use real malloc by ikegami (Patriarch) on Mar 15, 2011 at 19:11 UTC
Re: Make Perl use real malloc by Khen1950fx (Canon) on Mar 15, 2011 at 06:36 UTC
Whether the analysis seems sound or not, I'd probably not want to return to v5.004. If you configured perl with -DDEBUGGING, and you chose to use the perl malloc, then try this script: `#!/usr/bin/perl use strict; use warnings; warn('!'); do 'lib/auto/POSIX/autosplit.ix'; warn('!!! "after"');` [download] Run it like this: `perl -Dm script.pl` [download] I ran it with -Dm, but if you want other options, run it with -D for a complete list of options.	[reply] [d/l] [select]


Perl Monk, Perl Meditation
	PerlMonks