Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

Re: [RFC] Building Regex Alternations Dynamically

by LanX (Saint)
on Jan 18, 2017 at 13:56 UTC ( [id://1179850]=note: print w/replies, xml ) Need Help??


in reply to Building Regex Alternations Dynamically

it's almost TL;DR, so two separate posts my have been better. :)

Just three remarks:

  • IIRC there is a hard limit for the length of a regex-string ... (or was it only the repetition counter .{x} limited to x aprox 32000 ?)
  • "newer" Perls come with something called trie-optimization which will considerably speed up such or-clauses.
  • Trie-optimization has definitely a hard limit, where it falls back to (much slower) old behaviour.

Cheers Rolf
(addicted to the Perl Programming Language and ☆☆☆☆ :)
Je suis Charlie!

Replies are listed 'Best First'.
Re^2: [RFC] Building Regex Alternations Dynamically
by haukex (Archbishop) on Jan 18, 2017 at 14:03 UTC

    Hi LanX,

    Thanks for your comments, I'll think about whether I can provide a TL;DR version - my thought was that impatient users could just copy-n-paste my code :-)

    IIRC there is a hard limit for the length of a regex-string

    I'm not sure, I'll have to look into it, but I do know this works :-)

    my $word_regex = do { open my $wfh, '<:utf8', '/usr/share/dict/words' or die $!; chomp( my @words = <$wfh> ); close $wfh; my $re = join '|', map {quotemeta} sort { length $b <=> length $a or $a cmp $b } @words; qr/$re/i }; print length($word_regex), "\n"; __END__ 964836

    Thanks,
    -- Hauke D

      > > IIRC there is a hard limit for the length of a regex-string

      Maybe not in the string length but for sure in the processing.

      something like /x{33000}/ will result in

      Quantifier in {,} bigger than 32766 in regex; ...

      But that's not exactly relevant for your case :)

      Cheers Rolf
      (addicted to the Perl Programming Language and ☆☆☆☆ :)
      Je suis Charlie!

Re^2: [RFC] Building Regex Alternations Dynamically
by haukex (Archbishop) on Jan 21, 2017 at 17:09 UTC

    Hi Rolf,

    IIRC there is a hard limit for the length of a regex-string

    Other than RAM, I haven't found one yet, I just built a qr// regex from a 1GB text file without alternations and a second from 40MB of text with over 10M alternations.

    As you may have seen I added a TL;DR, thanks for the idea!

    Regards,
    -- Hauke D

      Yeah sorry I had

      > > (or was it only the repetition counter .{x} limited to x aprox 32000 ?)

      in mind. Since regexes are compiled to op codes hard limits are unlikely.

      > As you may have seen I added a TL;DR, thanks for the idea!

      Hmm probably it's time to include support for TOC = table of content in my wiki add on :)

      Cheers Rolf
      (addicted to the Perl Programming Language and ☆☆☆☆ :)
      Je suis Charlie!

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1179850]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (4)
As of 2024-04-19 13:13 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found