Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses

Re^2: Performance penalty of using qr//

by Anonymous Monk
on Dec 21, 2017 at 13:22 UTC ( #1205978=note: print w/replies, xml ) Need Help??

in reply to Re: Performance penalty of using qr//
in thread Performance penalty of using qr//

Um ... so ... when is the sometime the optimization kicks in? Can it be measured?
  • Comment on Re^2: Performance penalty of using qr//

Replies are listed 'Best First'.
Re^3: Performance penalty of using qr//
by dave_the_m (Monsignor) on Dec 21, 2017 at 13:29 UTC
    Um ... so ... when is the sometime the optimization kicks in
    What optimisation are you referring to?


      I'm guessing the optimization of using qr// over a plain string (ie: "Since Perl may compile the pattern at the moment of execution of the qr() operator, using qr() may have speed advantages in some situations ...")

      From your previous post I would say that this happens when there is a big compilation overhead, so I thought about this, and used this list of words for testing:

      use strict; use warnings; use Benchmark qw( cmpthese timethese ); open my $words, "<", "linuxwords.txt" or die "$!"; my @words = <$words>; chomp @words; my @search = @words[0..10]; $" = "|"; my $re = qr/^(?:@words)$/; my $str = "^(?:@words)\$"; my $r = timethese ( -5, { use_qr => sub { map /$re/, @search }, use_str => sub { map /$str/, @search }, use_re => sub { map /^(?:@words)$/, @search }, } ); cmpthese $r;
      Benchmark: running use_qr, use_re, use_str for at least 5 CPU seconds. +.. use_qr: 5 wallclock secs ( 5.23 usr + 0.00 sys = 5.23 CPU) @ 98 +736.51/s (n=515997) use_re: 5 wallclock secs ( 5.33 usr + 0.00 sys = 5.33 CPU) @ 23 +.99/s (n=128) use_str: 5 wallclock secs ( 5.23 usr + 0.00 sys = 5.23 CPU) @ 22 +68.47/s (n=11855) Rate use_re use_str use_qr use_re 24.0/s -- -99% -100% use_str 2268/s 9355% -- -98% use_qr 98737/s 411431% 4253% --
      The re case is pretty bad because of the systematic interpolation, but I can't help but feel like I might be missing something because of how absurd the difference between qr and str is? But if this is correct, then qr is a clear winner for dictionary search.

        You've created a benchmark where the cost of compiling a single regex containing 45000 alternations is overwhelmingly more expensive than running that compiled regex 11 times.

        In the qr case, the pattern is compiled once, *before* the benchmark is run. The benchmark cost is 11 matches, plus 11 clonings of the compiled regex's internal struture.

        In the str case, the benchmark includes compiling the pattern before matching the first word. For the subsequent 10 matches it attempt to recompile /$str/, but each time it uses an optimisation where it sees if the string has changed since last time and if so skips recompiling it.

        So sometimes /$str/ in a loop for unchanging $str can be faster than /$qr/, but that benchmark won't show it.

        Or more formally, if C is the time to compile a pattern, M is the time to run (match) against the compiled pattern, and D is the time to duplicate the regex structure, then

        $qr = /.../; /$qr/ for 1.N
        takes C + N(M+D), but your benchmark was measuring N(M+D); while
        $str = "..."; /$str/ for 1.N
        takes N(C+M) C + NM, which was what your benchmark was measuring. (I just corrected the above - I forgot to include the 'unchanged pattern' optimisation)


Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1205978]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others romping around the Monastery: (3)
As of 2021-04-10 12:33 GMT
Find Nodes?
    Voting Booth?

    No recent polls found