You could take a look at Devel::Size, Devel::Size::Report, Devel::Peek, Devel::FindGlobals and, if you can build your own copy of Perl for the target platforms, Devel::LeakTrace.

That said, to be able to generate a 1.7GB footprint from a script that processes a couple of 10 MB files requires that you are either building a lot of duplicate datastructures (hashes or arrays)--this is quite easy to do with several of the Graph::* and Set::* type modules; many of which are quite profligate with memory.

This is especially true if your building Graphs that result in self referencing trees and the like and don't have code to explicitely break circular references as these will prevent garbage collection.

The other way to chew up large volumes of ram unnecessarially, is to pass lots of bigs lists around between subroutines, rather than array references. The classic example is something like:

This consumes 390MB to process 10 MB of data:

Whereas this does the same processing and in 170 MB by using references and side-effects to avoiding duplicating large lists

And this does the same using less than 2 MB by avoiding building large lists in the first place:

perl -nlwe" m[s*(\S+)\s*] and print $1*2" data\1millionlines.dat
[download]

It's a contrived example, but it illustrates the points.

Using my to tell Perl when you don't need a variable anymore is a good start, but Perl uses reference counting for memory management, so you might run into circular references like the following:

my $bar = {};
my $foo = { bar => \$bar };
$bar->{foo} = \$foo;
[download]

Here, $bar points to $foo and reverse, so there is no way that Perl will ever release the two variables back until the process exits.

There are a number of solutions to circumvent the problem. The easiest way is to split up your program into several programs, which all process a small amount of the data and then exit, thus cleaning up the memory.

There is also weaken in Scalar::Util, which creates a weak reference, which will not prevent the target from being collected. To know which references to weaken is not easy though.

You can also try to analyze your code and find out where circular references are, and manually break them, by setting in the example $foo->{bar} = undef.

A second thing might be that your program is simply creating too complex memory structures - for example, I can well imagine an 10 MB XML file eating up lots of memory - so you might want to consider changing your data structures or changing from a DOM XML parser to an event based parser. Such restructuring is not easy though.

One especially weird aspect was that if I ran my program in the debugger, the memory leaks disappeared.

But, it only happened on Solaris 2.7 and not 2.8. It seems there the system memory management (by Solaris) was flawed, and it was fixed for 2.8. Sorry, I don't remember the details.

Maybe this is the problem??

Good Luck

Sandy

UPDATE: Another related node Managing System Memory Resources