|
in reply to Re: Setting end position for the regexp engine
in thread Setting end position for the regexp engine
BrowserUK thank you very much for the answer!
I have made two new versions of the example parser using the LVALUE returned by substr.
In version 2 I use [ $type, $first, $length ] to represent a part in @part.
In version 3 I use [ $type, $sub_txt_ref ], where $sub_txt_ref is a LVALUE returned by substr.
The scalar type LVALUE is new form me. Where can I read about it?
Example of parser version 2
use strict;
use warnings;
sub parse {
my ( $txt_ref, $part_aref, $spec_aref ) = @_;
if ( not defined $part_aref ) { # use the whole $$txt_ref
$part_aref = [ [ '??', 0, length($$txt_ref) ] ];
}
my @part; # result from the parsing
for my $part (@$part_aref) {
my ( $type, $first, $length ) = @$part;
if ( $type ne '??' ) {
push @part, $part;
next;
}
my $sub_txt_ref = \substr( $$txt_ref, $first, $length );
my $end_last_match = 0;
MATCH:
{
for my $spec (@$spec_aref) {
my ( $reg_exp, $type ) = @$spec;
if ( $$sub_txt_ref =~ m{$reg_exp}gcm ) {
push @part, [ $type, $-[0] + $first, $+[0] - $-[0]
+ ];
$end_last_match = $+[0];
redo MATCH;
}
}
}
if ( $end_last_match < $length ) {
warn 'ERROR: Stopped before string end at pos: ',
"$first + $end_last_match\n<",
substr( $$sub_txt_ref, $end_last_match ), '>';
}
}
return \@part;
}
my @spec_1 = (
[ qr{\G'[^'\\]*(?:\\.[^'\\]*)*'}, 'sq' ],
[ qr{\G"[^"\\]*(?:\\.[^"\\]*)*"}, 'dq' ],
[ qr{\G//[^\n]*[\n]?}, 'cn' ],
[ qr{\G/[*](?:[^*]*|[*]+[^/*]*)*[*]/}, 'cb' ],
[ qr{\G(?:[^'"/]+|[/][^'"/*])+}, '??' ],
);
my $dig1 = qr{-?\d+\.\d*};
my $dig2 = qr{-?\d*\.\d+};
my $dig3 = qr{-?\d+};
my $dig4 = qr{E-?\d+};
my $digit = qr{(?:$dig1|$dig2|$dig3)$dig4?};
my @spec_2 = (
[ qr{\G(?:var|alert)}, 'rw' ],
[ qr{\G$digit}, 'di' ],
[ qr{\G[_a-zA-Z0-9.\$]+}, 'na' ],
[ qr{\G(?:[^_a-zA-Z0-9.\$\d]+|[\n\s]+)+}, '??' ],
);
sub to_string_part_aref {
my ( $txt_ref, $part_aref ) = @_;
return join '', map {
my ( $type, $first, $length ) = @$_;
if ( $type eq '??' ) {
substr( $$txt_ref, $first, $length );
}
else {
"<$type>" . substr( $$txt_ref, $first, $length ) . "</$typ
+e>";
}
} @{$part_aref};
}
my $text = do { local $/; <DATA> };
my $text_ref = \$text;
my $part_ref_1 = parse( $text_ref, undef, \@spec_1 );
my $part_ref_2 = parse( $text_ref, $part_ref_1, \@spec_2 );
warn to_string_part_aref( $text_ref, $part_ref_2 );
__DATA__
// This is a single-line comment
var x = 4; // Single
/* Multiple-line
comment that can
span any number
of lines */
/*
This is a multi-line comment
// Still a multi-line comment
*/
/* Stop code
var x = 4;
var y = 5;
/* Bug? *
x = "cool";
End Stop code */
// This is a single-line comment /* ...still a single-line comment
'string\' // still a string'; // comment /* not-a-nested-comment
var = 0.5; // comment */* still-a-comment
' /**/ string ' /* "comment..."
// still-a-comment */ alert('This isn\'t a comment!');
/\/* this isn't a comment! */; //* comment
/*
//a comment... // still-a-comment
12345
"Foo /bar/ ""
*/
/*//Boo*/
/*/**/
Example of parser version 3
Is updated! Removed the parameter $txt_ref.
use strict;
use warnings;
sub first_part_aref {
my ($txt_ref) = @_;
my $sub_txt_ref = \substr( $$txt_ref, 0 );
return [ [ '??', $sub_txt_ref ] ];
}
sub parse {
my ( $part_aref, $spec_aref ) = @_;
my @part; # result from the parsing
for my $part (@$part_aref) {
my ( $type, $sub_txt_ref ) = @$part;
if ( $type ne '??' ) {
push @part, $part;
next;
}
my $end_last_match = 0;
MATCH:
{
for my $spec (@$spec_aref) {
my ( $reg_exp, $type ) = @$spec;
if ( $$sub_txt_ref =~ m{$reg_exp}gcm ) {
my $sub_sub_txt_ref =
\substr( $$sub_txt_ref, $-[0], $+[0] - $-[0] );
push @part, [ $type, $sub_sub_txt_ref ];
$end_last_match = $+[0];
redo MATCH;
}
}
}
if ( $end_last_match < length $$sub_txt_ref ) {
warn 'ERROR: Stopped before string end at pos: ',
"$end_last_match\n<",
substr( $$sub_txt_ref, $end_last_match ), '>';
}
}
return \@part;
}
my @spec_1 = (
[ qr{\G'[^'\\]*(?:\\.[^'\\]*)*'}, 'sq' ],
[ qr{\G"[^"\\]*(?:\\.[^"\\]*)*"}, 'dq' ],
[ qr{\G//[^\n]*[\n]?}, 'cn' ],
[ qr{\G/[*](?:[^*]*|[*]+[^/*]*)*[*]/}, 'cb' ],
[ qr{\G(?:[^'"/]+|[/][^'"/*])+}, '??' ],
);
my $dig1 = qr{-?\d+\.\d*};
my $dig2 = qr{-?\d*\.\d+};
my $dig3 = qr{-?\d+};
my $dig4 = qr{E-?\d+};
my $digit = qr{(?:$dig1|$dig2|$dig3)$dig4?};
my @spec_2 = (
[ qr{\G(?:var|alert)}, 'rw' ],
[ qr{\G$digit}, 'di' ],
[ qr{\G[_a-zA-Z0-9.\$]+}, 'na' ],
[ qr{\G(?:[^_a-zA-Z0-9.\$\d]+|[\n\s]+)+}, '??' ],
);
sub to_string_part_aref {
my ($part_aref) = @_;
return join '', map {
my ( $type, $sub_txt_ref ) = @$_;
$type ne '??' ? "<$type>" : (),
$$sub_txt_ref,
$type ne '??' ? "</$type>" : ();
} @{$part_aref};
}
my $text = do { local $/; <DATA> };
my $part_ref_0 = first_part_aref( \$text );
my $part_ref_1 = parse( $part_ref_0, \@spec_1 );
my $part_ref_2 = parse( $part_ref_1, \@spec_2 );
warn to_string_part_aref($part_ref_2);
__DATA__
// This is a single-line comment
var x = 4; // Single
/* Multiple-line
comment that can
span any number
of lines */
/*
This is a multi-line comment
// Still a multi-line comment
*/
/* Stop code
var x = 4;
var y = 5;
/* Bug? *
x = "cool";
End Stop code */
// This is a single-line comment /* ...still a single-line comment
'string\' // still a string'; // comment /* not-a-nested-comment
var = 0.5; // comment */* still-a-comment
' /**/ string ' /* "comment..."
// still-a-comment */ alert('This isn\'t a comment!');
/\/* this isn't a comment! */; //* comment
/*
//a comment... // still-a-comment
12345
"Foo /bar/ ""
*/
/*//Boo*/
/*/**/
|
Re^3: Setting end position for the regexp engine using LVALUE
by ikegami (Patriarch) on Sep 19, 2013 at 23:31 UTC
|
ref returns LVALUE for a reference to a PVLV, a scalar subtype. You can read about it a bit here. It's basically a scalar that's going to have magic associated with it, and it has a few extra fields for storage the magic can use.
Magic, among other things, allows one to attach a getter and fetcher to a variable.
You might think that
substr($_, 3, 2) = 'foo';
gets converted to
substr($_, 3, 2, 'foo');
by the compiler, but it isn't. substr is truly called as is. This allows the following to work
my $ref = \substr($_, 3, 2);
$$ref = 'foo';
That means substr must return a magical scalar.
There are more than one type of scalar. Some can hold an integer, some can hold a string, some can hold both. Perl automatically upgrades a scalar when necessary.
Magical scalars require extra fields to store information about what magic is attached to the scalar. The most basic scalar subtype capable of being magical is the PVMG, but there is also PVLV. The PVLV is a PVMG with four extra fields: TYPE, TARGOFF, TARGLEN and TARG.
A instance of substr in an lvalue context returns a PVLV. It uses TARG, TARGOFF and TARGLEN to store the three arguments passed to substr[1].
>perl -MDevel::Peek -e"$_ = 'abcdef'; my $ref = \substr($_, 3, 2); Dum
+p($$ref);"
SV = PVLV(0x4d3d24) at 0x2cb29c <--- PVLV
REFCNT = 1
FLAGS = (TEMP,GMG,SMG) <--- "get" magic and "set" magic
IV = 0
NV = 0
PV = 0
MAGIC = 0x49f09c
MG_VIRTUAL = &PL_vtbl_substr <--- Function pointers for magic
MG_TYPE = PERL_MAGIC_substr(x) <--- x = substr magic
TYPE = x <--- x = substr magic
TARGOFF = 3 <--- For "x", start offset of substring
TARGLEN = 2 <--- For "x", length of substring
TARG = 0x4a8414 <--- For "x", addr of associated scalar ($_)
FLAGS = 0
SV = PV(0x2c8a6c) at 0x4a8414 <--- Dump of associated scalar ($_)
REFCNT = 2
FLAGS = (POK,pPOK)
PV = 0x2cab94 "abcdef"\0
CUR = 6
LEN = 12
When you try to fetch from $$ref, the associated get magic first does something like $$ref = substr($$TARG, $TARGOFF, $TARGLEN);.
After you assign to $$ref, the associated "set" magic effectively does substr($$TARG, $TARGOFF, $TARGLEN, $$ref);.
Notes
- Missing and negative arguments are resolved before being assigned.
| [reply]
[d/l]
[select] |
Re^3: Setting end position for the regexp engine using LVALUE
by BrowserUk (Patriarch) on Sep 19, 2013 at 07:54 UTC
|
The scalar type LVALUE is new form me. Where can I read about it?
Um. Beyond perldoc:substr, I'm not sure.
It has been a part of my lexicon for so long, I've forgotten how or when I first learnt of it.
And of course there is this, where the hereditary peer hereabouts ... (draw your own conclusions).
With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
| [reply] |
|
|
