Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number
 
PerlMonks  

Re^2: Lexing C++

by Random_Walk (Prior)
on Sep 02, 2019 at 19:51 UTC ( #11105468=note: print w/replies, xml ) Need Help??


in reply to Re: Lexing C++
in thread Lexing C++

Hi Eily,

Thanks for your comments, very useful. I know the code I am looking at makes use of C templates quite a lot, so hopefully there are no evil compiler tricks to trip me up.

For multi-line comments, strings and such, I was going to rely on my parser using a state machine so it knows when it is inside such a beast. In fact this sort of thing is why I am trying to lex and parse. I had a previous solution mostly based on regex, but handling multi-line strings, comments etc, was one of the issues that was making it hairy and un-maintainable.

I certainly need to improve my lexer to handle '<<' and friends, the previous comment also highlighted that fact. I think I need to break on \s+ first to better get identifiers all together, then perhaps \b to get multi character tokens, finally per character to get the }); sort of fun. An update is in the works...

Cheers,
R.

Pereant, qui ante nos nostra dixerunt!

Replies are listed 'Best First'.
Re^3: Lexing C++
by tybalt89 (Prior) on Sep 02, 2019 at 21:18 UTC

    Here's a slightly modified version (still incomplete) of mine that handles several of the problems you mention, like multi-line comments and strings (updated per afoken Re^2: Lexing C++). It also adds the character position of the token as the third item, so the parser can generate better error messages :)

    #!/usr/bin/perl # https://perlmonks.org/?node_id=11105353 # following spirit of my http://www.rosettacode.org/wiki/Compiler/lexi +cal_analyzer#Alternate_Perl_Solution use strict; use warnings; my @tokens; my %reserved = map { $_ => 'reserved' } qw( alignas alignof and and_eq asm atomic_cancel atomic_commit atomic_noexcept auto bitand bitor bool break case catch char char16_t char32_t class compl concept const constexpr const_cast continue co_await co_return co_yield decltype default delete do double dynamic_cast else enum explicit export extern false float for friend goto if import inline int long module mutable namespace new noexcept not not_eq nullptr operator or or_eq private protected public register reinterpret_cast requires return short signed sizeof static static_assert static_cast struct switch synchronized template this thread_local throw true try typedef typeid typename union unsigned using virtual void volatile wchar_t while xor xor_eq ); my %Ops = ( # Single or multiple operators by name '(' => 'LeftParen', ')' => 'RightParen', '[' => 'LeftSquare', ']' => 'RightSquare', '{' => 'LeftCurly', '}' => 'RightCurly', '<' => 'LessThan', '>' => 'GreaterThan', '=' => 'Equal', '+' => 'Plus', '-' => 'Minus', '*' => 'Asterisk', '/' => 'Slash', '#' => 'Hash', '.' => 'Dot', ',' => 'Comma', ':' => 'Colon', ';' => 'Semicolon', "'" => 'SingleQuote', '"' => 'DoubleQuote', '|' => 'Pipe', '>>' => 'RightShift', # remember to sort by longest first '<<' => 'LeftShift', '<=' => 'LessThanOrEqual', '>=' => 'GreaterThanOrEqual', '||' => 'LogicalOr', '&&' => 'LogicalAnd', '+=' => 'PlusEqual', '-=' => 'MinusEqual', '*=' => 'TimesEqual', '/=' => 'DivideEqual', ); my $matchops = qr/(?:@{[ join '|', map quotemeta, sort { length $b <=> length $a } # longest first sort keys %Ops ]})/; my $regex = qr/ \G (?| \s+ (?{ undef }) | \/\/.* (?{ undef }) | \/\*[\s\S]*?\*\/ (?{ undef }) # assuming non-nested | \#(.+) (?{ [ 'Directive', $1 ] }) | \d+(?:\.\d*)? (?{ 'Number' }) | \.\d+ (?{ 'Number' }) | \w+ (?{ $reserved{$&} or 'Identifier' }) | "((?:\\.|[^\\\n"])*)" (?{ [ 'string', $1 =~ s!\\(.)!$1!gr ] }) | '([^\\'\n])' (?{ [ 'Number', ord $1 ] }) | (?<!:)::(?!:) (?{ 'dblColon' }) | $matchops (?{ $Ops{$&} }) | . (?{ 'ERROR: unexpected character' }) ) /x; $_ = (join '', <DATA>) =~ s/\\\n/ /gr; defined $^R and push @tokens, [ ref $^R ? @{$^R} : ( $^R, $& ), $-[0] +] while /$regex/g; use Data::Dump 'dd'; dd @tokens; __DATA__ #define TheAnswerToLifeTheUniverseAndEverything \ (42) int main(int argc, char *argv[ ]) { int $foo = 1 << 5; /* multiline comment */ puts("testing a \"quoted\" string with $ sign"); exit(0); // success }

    Outputs:

    ( [ "Directive", "define TheAnswerToLifeTheUniverseAndEverything \t(42)", 0, ], ["reserved", "int", 55], ["Identifier", "main", 59], ["LeftParen", "(", 63], ["reserved", "int", 64], ["Identifier", "argc", 68], ["Comma", ",", 72], ["reserved", "char", 74], ["Asterisk", "*", 79], ["Identifier", "argv", 80], ["LeftSquare", "[", 84], ["RightSquare", "]", 86], ["RightParen", ")", 87], ["LeftCurly", "{", 90], ["reserved", "int", 93], ["ERROR: unexpected character", "\$", 97], ["Identifier", "foo", 98], ["Equal", "=", 102], ["Number", 1, 104], ["LeftShift", "<<", 106], ["Number", 5, 109], ["Semicolon", ";", 110], ["Identifier", "puts", 140], ["LeftParen", "(", 144], ["string", "testing a \"quoted\" string with \$ sign", 145], ["RightParen", ")", 186], ["Semicolon", ";", 187], ["Identifier", "exit", 190], ["LeftParen", "(", 194], ["Number", 0, 195], ["RightParen", ")", 196], ["Semicolon", ";", 197], ["RightCurly", "}", 211], )

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://11105468]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (8)
As of 2021-04-22 17:31 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?