Here's a slightly modified version (still incomplete) of mine that handles several of the problems you mention,
like multi-line comments and strings (updated per afoken Re^2: Lexing C++).
It also adds the character position of the token as the third item, so the parser can generate better error messages :)
#!/usr/bin/perl
# https://perlmonks.org/?node_id=11105353
# following spirit of my http://www.rosettacode.org/wiki/Compiler/lexi
+cal_analyzer#Alternate_Perl_Solution
use strict;
use warnings;
my @tokens;
my %reserved = map { $_ => 'reserved' } qw(
alignas alignof and and_eq asm atomic_cancel atomic_commit
atomic_noexcept auto bitand bitor bool break case catch char
char16_t char32_t class compl concept const constexpr const_cast
continue co_await co_return co_yield decltype default delete do
double dynamic_cast else enum explicit export extern false float for
friend goto if import inline int long module mutable namespace new
noexcept not not_eq nullptr operator or or_eq private protected
public register reinterpret_cast requires return short signed
sizeof static static_assert static_cast struct switch synchronized
template this thread_local throw true try typedef typeid typename
union unsigned using virtual void volatile wchar_t while xor xor_eq
);
my %Ops = ( # Single or multiple operators by name
'(' => 'LeftParen',
')' => 'RightParen',
'[' => 'LeftSquare',
']' => 'RightSquare',
'{' => 'LeftCurly',
'}' => 'RightCurly',
'<' => 'LessThan',
'>' => 'GreaterThan',
'=' => 'Equal',
'+' => 'Plus',
'-' => 'Minus',
'*' => 'Asterisk',
'/' => 'Slash',
'#' => 'Hash',
'.' => 'Dot',
',' => 'Comma',
':' => 'Colon',
';' => 'Semicolon',
"'" => 'SingleQuote',
'"' => 'DoubleQuote',
'|' => 'Pipe',
'>>' => 'RightShift', # remember to sort by longest first
'<<' => 'LeftShift',
'<=' => 'LessThanOrEqual',
'>=' => 'GreaterThanOrEqual',
'||' => 'LogicalOr',
'&&' => 'LogicalAnd',
'+=' => 'PlusEqual',
'-=' => 'MinusEqual',
'*=' => 'TimesEqual',
'/=' => 'DivideEqual',
);
my $matchops = qr/(?:@{[ join '|', map quotemeta,
sort { length $b <=> length $a } # longest first
sort keys %Ops ]})/;
my $regex = qr/ \G (?|
\s+ (?{ undef })
| \/\/.* (?{ undef })
| \/\*[\s\S]*?\*\/ (?{ undef }) # assuming non-nested
| \#(.+) (?{ [ 'Directive', $1 ] })
| \d+(?:\.\d*)? (?{ 'Number' })
| \.\d+ (?{ 'Number' })
| \w+ (?{ $reserved{$&} or 'Identifier' })
| "((?:\\.|[^\\\n"])*)" (?{ [ 'string', $1 =~ s!\\(.)!$1!gr ] })
| '([^\\'\n])' (?{ [ 'Number', ord $1 ] })
| (?<!:)::(?!:) (?{ 'dblColon' })
| $matchops (?{ $Ops{$&} })
| . (?{ 'ERROR: unexpected character' })
) /x;
$_ = (join '', <DATA>) =~ s/\\\n/ /gr;
defined $^R and push @tokens, [ ref $^R ? @{$^R} : ( $^R, $& ), $-[0]
+]
while /$regex/g;
use Data::Dump 'dd'; dd @tokens;
__DATA__
#define TheAnswerToLifeTheUniverseAndEverything \
(42)
int main(int argc, char *argv[ ])
{
int $foo = 1 << 5;
/* multiline
comment */
puts("testing a \"quoted\" string with $ sign");
exit(0); // success
}
Outputs:
(
[
"Directive",
"define TheAnswerToLifeTheUniverseAndEverything \t(42)",
0,
],
["reserved", "int", 55],
["Identifier", "main", 59],
["LeftParen", "(", 63],
["reserved", "int", 64],
["Identifier", "argc", 68],
["Comma", ",", 72],
["reserved", "char", 74],
["Asterisk", "*", 79],
["Identifier", "argv", 80],
["LeftSquare", "[", 84],
["RightSquare", "]", 86],
["RightParen", ")", 87],
["LeftCurly", "{", 90],
["reserved", "int", 93],
["ERROR: unexpected character", "\$", 97],
["Identifier", "foo", 98],
["Equal", "=", 102],
["Number", 1, 104],
["LeftShift", "<<", 106],
["Number", 5, 109],
["Semicolon", ";", 110],
["Identifier", "puts", 140],
["LeftParen", "(", 144],
["string", "testing a \"quoted\" string with \$ sign", 145],
["RightParen", ")", 186],
["Semicolon", ";", 187],
["Identifier", "exit", 190],
["LeftParen", "(", 194],
["Number", 0, 195],
["RightParen", ")", 196],
["Semicolon", ";", 197],
["RightCurly", "}", 211],
)
-
Are you posting in the right place? Check out Where do I post X? to know for sure.
-
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big>
<blockquote> <br /> <dd>
<dl> <dt> <em> <font>
<h1> <h2> <h3> <h4>
<h5> <h6> <hr /> <i>
<li> <nbsp> <ol> <p>
<small> <strike> <strong>
<sub> <sup> <table>
<td> <th> <tr> <tt>
<u> <ul>
-
Snippets of code should be wrapped in
<code> tags not
<pre> tags. In fact, <pre>
tags should generally be avoided. If they must
be used, extreme care should be
taken to ensure that their contents do not
have long lines (<70 chars), in order to prevent
horizontal scrolling (and possible janitor
intervention).
-
Want more info? How to link
or How to display code and escape characters
are good places to start.