Beware List::MoreUtils natatime()!

I just got done tracking down a nasty memory leak due to my use of natatime(). It's not a function I've ever used before but it seemed convenient enough.

Run this code and watch the memory use grow:

use List::MoreUtils qw(natatime);

my $count = 0;
while (1) {
    my @rows = ("...") x 100;
    my $it = natatime(10,@rows);
    while(my @chunk = $it->()) {
        $count += @chunk;
        if ($count % 1000 == 0) {
            warn "COUNT : $count\n";
        }
    }
}
[download]

I haven't dug any deeper than this, so I'm not sure if it's something in List::MoreUtils XS code or a side-effect of using a closure for the iterator, or what. I'm also going to send this to the maintainer, so no need to remind me!

UPDATE: It would seem to be an XS bug - setting the environment var LIST_MOREUTILS_PP=1 makes it go away.

UPDATE 2: Corion asked, so I'll tell you - this is List::MoreUtils v0.22 on Perl v5.8.8. Oh, and the maintainer's email address doesn't work. Anybody want to take over and fix this?

-sam

Comment on Beware List::MoreUtils natatime()! Download Code

Replies are listed 'Best First'.
Re: Beware List::MoreUtils natatime()! by moritz (Cardinal) on Feb 09, 2009 at 09:21 UTC
Others who are interested in List::MoreUtils' bugs will probably look at the bug tracker, so maybe you should open a ticket there. And FYI, there are already two other reports about memory leaks in that queue.	[reply]
Re: Beware List::MoreUtils natatime()! by JavaFan (Canon) on Feb 09, 2009 at 10:17 UTC
It would seem to be an XS bug - setting the environment var LIST_MOREUTILS_PP=1 makes it go away. Not for me: $ LIST_MOREUTILS_PP=1 perl foo String found where operator expected at (eval 1) line 206, near "Carp: +:croak "each_arrayref: argument is not an array reference\n"" (Do you need to predeclare Carp::croak?) String found where operator expected at (eval 1) line 226, near "Carp: +:croak "each_array: unknown argument '$method' passed to iterator."" (Do you need to predeclare Carp::croak?) Use of inherited AUTOLOAD for non-method List::MoreUtils::natatime() i +s deprecated at foo line 14. Can't locate auto/List/MoreUtils/natatime.al in @INC (@INC contains: / +opt/perl/lib/5.10.0/i686-linux-64int-ld /opt/perl/lib/5.10.0 /opt/per +l/lib/site_perl/5.10.0/i686-linux-64int-ld /opt/perl/lib/site_perl/5. +10.0 /opt/perl/lib/site_perl/5.8.8 /opt/perl/lib/site_perl .) at foo +line 14 [download] It runs (with the memory leak) if I don't set LIST_MOREUTILS_PP.	[reply] [d/l]
Re^2: Beware List::MoreUtils natatime()! by diotalevi (Canon) on Feb 10, 2009 at 07:13 UTC
Your bug sounds like your version is depending on an accident of not-quite recent perls. It used to often "just work" to say `Carp::croak '...';`. That was accidental and whenever you see an exception like this, just make sure to load Carp in advance of loading the thing you're using. In your example, do: `$ LIST_MOREUTILS_PP=1 perl -MCarp foo`	[reply] [d/l] [select]
Re^3: Beware List::MoreUtils natatime()! by JavaFan (Canon) on Feb 10, 2009 at 07:55 UTC
Looking into the source of List::MoreUtils, I find: `require Carp; Carp::croak "some message";` [download] That only works if Carp was already loaded. In older versions of Perl, there was a common pragma (vars?) that loaded it, but that no longer is loaded in 5.10. Hence the error. And another reason to not use List::MoreUtils.	[reply] [d/l]
Re^4: Beware List::MoreUtils natatime()! by diotalevi (Canon) on Feb 10, 2009 at 08:03 UTC
Re^5: Beware List::MoreUtils natatime()! by jdporter (Paladin) on Feb 10, 2009 at 11:48 UTC


more useful options
	PerlMonks