that's a heuristic you need to work out.

Actually, there was a bug, which ikegami quickly fixed. In the original version of his lexing code, the first two alternative patterns matched every possible valid, non-empty string, making the remaining two alternative patterns unreachable.

if the alternatives are not exclusive, then the leftest match will always match first.

I explained the options are mutually exclusive in my original post. It's important that each alternative pattern matches one and only one class of token.

i believe you that the options are mutually exclusive, but i never program as such. it's difficult to guarantee non-trivial regexes will indeed match exclusively on all input data. and hence i would either pre-run all regexes or other such programming to eliminate non-exclusivity OR allow all regexes to match against input and make normal logic decisions + sane decisions on possible anomalies. that's the kind of defensive programming i'd do if time allows. it's always a balancing act in handling program input, but i think a little scepticism in programming itself is a good thing. hence my advice...even though it's not inline with the spec. take it or leave it as you please.
the hardest line to type correctly is: stty erase ^H