You might find Applying regexes to streams: Perl enhancement idea informative.

Yes, the /z modifier would precisely fit what I'm looking for.

Let me clarify my problem by contrasting it with your sample code:

• The stream should consist exclusively of a sequence of matchable texts. That is, $` should always be ''. (Remember the ^ at the beginning of my $pattern.)
• I want to know as soon as possible that a pattern won't be able to match the buffer. It's somewhat easier for me because I know that my pattern begins with ^.

Dare I open up the perl source code and try to implement this now?

I poked around in regexec.c. Whoa, heavy magic! Maybe --just maybe-- if the perl RE engine weren't so optimized, I would have the confidence to change stuff and implement /z. But not on the existing RE engine.

I'm now considering using Sandfly's suggested technique with YAPE::Regexp to provide the list of atoms.

Thanks to all for the useful comments and suggetions.

my @re= qw( [abc] b+ c );

my @tests = qw( a ab ba bb bbbbbcdef bbbbbbg );

TEST:
foreach my $buff ( @tests )
{
    my @tre = @re;
    my $tre = "";
    while (@tre > 1)
    {
        $tre .= shift @tre;
        if ( $buff =~ m/^$tre$/ )
        {
            print "Test string '$buff' matched partial pattern, $tre\n
+";
            next TEST;
        }
    }
    $tre .= shift @tre;
    if ( $buff =~ m/^$tre/ )
    {
        print "Test string '$buff' matched whole pattern, $tre\n";
    }
    else
    {
        print "Test string '$buff' did not match pattern $tre\n";
    }
}
[download]

This method will fail if you use alternation, or atoms with a minumum length > 1, such as "a{5}".

Ahh, not a horrible idea for simple patterns. I would need to add a ^ at the beginning and maintain a $ at the end of the RE. For a simple enough pattern I could even generate the partial REs programatically. Alas, it would be tougher to deal with quantifiers, capturing groups, etc, and, alas, I am quite the fan of quantifiers.

(It took me a while to recognize what you were suggesting. I didn't notice the ".=" in $tre .= shift @tre.)

If you know the fixed length of the buffer, simply either create a fixed length version of your regexp or work out manually, if you can, all the ways of expanding the regexp out to the length of the buffer. For fixed strings, use  substr. This will be much, much easier, that trying to interface to the perl regexp code.

-Mark

(I am using terms "parser", "tokenizer / lexer", "abstract syntax tree (AST)" in a compiler context.)

I was pointed to the YAPE::Regex module "Yet Another Parser/Extractor for Regular Expressions". From the documentation and my experiments in the debugger, I believe it to be only a lexer, not a proper parser. As far as I can tell, it breaks down the text of a RE into strings of characters with particular meanings (tokens), but it doesn't assemble those meanings into a hierarchy (abstract syntax tree).

For a quick look at the AST of a RE, you can see the indented text in "Debugging regular expressions" of man perldebguts or

perl
-Mre=debug -e '$re =  qr/^a(b(cd?)?)?/;'
[download]

.

To do one of the transformations that many of us have thought up, I should be operating on the AST, not just on the individual tokens. Also, I would have to break each EXACT node into a sequence of single-character EXACT nodes.