Substitution with regex and memory consumption

k-mx has asked for the wisdom of the Perl Monks concerning the following question:

Hello, comrades monks!

I want to share knowledge about interesting regex behavior:

#!/usr/bin/env perl

use strict;
use warnings;

use 5.022;

# WARNING! Will consume about 3GB of RAM
$| = 1;

my $size = 1024 * 1024 * 1000;
my $s    = 'C' x $size;

# eval << 'EOD';

$s =~ s/C/1/; # copy of CCCCC... will stay in memory
$s =~ s/C/2/; # same but 1CCCC...

# EOD

print "Time to measure memory usage...\n";

sleep(9000);
[download]

Every expression with substitution call, will copy original string and store it until next call. So, memory not leaking and will be reused (for e.g. inside loop blocks), but this behavior can lead to serious RAM consumption. The only known (for me) workaround for this problem is string eval, that will force memory reclaim, but this is clumsy in my opinion.

What do you think about such behavior? Is there more elegant way to free up memory used by 's///'?

UPD:

Reproduced on 5.22.2 (Slackware 14.2), 5.30.0 (Centos 7), and no memory waste on 5.16.3 (Centos 7)

Investigation with 'Test::LeakTrace'

Interesting links:

Strange memory growth? (can be handled with 'undef $foo')
[Lightning Talk] Cees Hek - "To undef or not to undef"
Forcing perl to garbage collect (not much related, but can be interesting)

Comment on Substitution with regex and memory consumption Select or Download Code

Replies are listed 'Best First'.
Re: Substitution with regex and memory consumption by dave_the_m (Monsignor) on Feb 29, 2020 at 20:27 UTC
In general after a successful match (or the match part of a substitution), the regex engine keeps a copy of the original string so that it can dynamically generate values for $1, $2, $&, $` etc on demand. This string needs to be kept for at least as long as the surrounding scope - i.e. the scope of $1 etc. The details are far more complex, but internally perl's Copy-on-Write mechanism often (but not always) avoids having to do a real copy. But it doesn't always work out for the most efficient use of memory. Dave.	[reply]
Re^2: Substitution with regex and memory consumption by k-mx (Scribe) on Mar 01, 2020 at 09:19 UTC
Okay, thank you! Some assumptions, please correct me if i'm wrong: Prior 5.18.0, s/// will copy original only if one of these was set: $&, $`, $'. Interpreter will set global PL_sawampersand flag that can't be disabled later. m// is also affected by this flag. Between 5.18.0 and 5.20.0, Perl can track usage of mentioned variables separately and copy only requested part of string. Perl 5.20.0+, successful s/// match always changes string, so COW mechanism always had to copy original. So, before 5.20 we have choice: avoid $&, $`, $' and use /p modifier to explicitly copy ${^*MATCH}. Now we can use $&, $`, $', m// don't suffer from PL_sawampersand anymore, but s/// will always copy original string, PL_sawampersand state doesn't matter, and nothing we can do with that.	[reply]
Re^3: Substitution with regex and memory consumption by dave_the_m (Monsignor) on Mar 01, 2020 at 22:47 UTC
That's roughly it, yes. Dave.	[reply]
Re^4: Substitution with regex and memory consumption by k-mx (Scribe) on Mar 08, 2020 at 08:16 UTC
Re^5: Substitution with regex and memory consumption by dave_the_m (Monsignor) on Mar 08, 2020 at 11:51 UTC


The stupid question is the question not asked
	PerlMonks