that's a heuristic you need to work out.

Actually, there was a bug, which ikegami quickly fixed. In the original version of his lexing code, the first two alternative patterns matched every possible valid, non-empty string, making the remaining two alternative patterns unreachable.

if the alternatives are not exclusive, then the leftest match will always match first.

I explained the options are mutually exclusive in my original post. It's important that each alternative pattern matches one and only one class of token.