Here's a reduced (for brevity of the output), modified (to do more different things) version of the code in the article.

use strict;
my $sql = <<'--SQL--';
select case
  when a=b then 'c'
  else 'd'
  end "tough_one",
  e as "even tougher"
from mytable
--SQL--

use HOP::Lexer 'make_lexer';
my @sql   = $sql;
my $lexer = make_lexer(
    sub { shift @sql },  # iterator
    [ WORD      => qr/\w+/i        ],
    [ DQWORD    => qr/"\w+"/       ],
    [ DQUOTED   => qr/"[^"]+"/     ],
    [ QUOTED    => qr/'[^']*'/     ],
    [ COMMA     => qr/,/           ],
    [ SPACE     => qr/\s+/, sub {} ],
);

# parse
my @out;
push @out, $_ while $_ = $lexer->();

# Data::Dump the output (elaborated in order to produce compact result
+s)
use Data::Dumper;
$Data::Dumper::Indent = 0;
$Data::Dumper::Terse = 1;
($\, $,) = ("\n", ",\n");
print map { Dumper $_ } @out;
[download]

['WORD','select'],
['WORD','case'],
['WORD','when'],
['WORD','a'],
'=',
['WORD','b'],
['WORD','then'],
'\'',
['WORD','c'],
'\'',
['WORD','else'],
'\'',
['WORD','d'],
'\'',
['WORD','end'],
'"',
['WORD','tough_one'],
'"',
['COMMA',','],
['WORD','e'],
['WORD','as'],
'"',
['WORD','even'],
['WORD','tougher'],
'"',
['WORD','from'],
['WORD','mytable']
[download]

['WORD','select'],
['WORD','case'],
['WORD','when'],
['WORD','a'],
'=',
['WORD','b'],
['WORD','then'],
['QUOTED', '\'c\''],
['WORD','else'],
['QUOTED','\'d\''],
['WORD','end'],
['DQWORD','"tough_one"'],
['COMMA',','],
['WORD','e'],
['WORD','as'],
['DQUOTED','"even tougher"'],
['WORD','from'],
['WORD','mytable']
[download]

So, why is it not doing what I want?

Update for the people who are too impatient to read the whole thread, I'll now reveal the solution to the mystery (thanks to cmarcelo): HOP::Lexer is not trying to find the leftmost matching token, unlike what the rest of the world tends to do. It tries to match the most important type of token first, and then tries to find the other types of tokens in what remains on its left. It never backtracks. So, you always should put the rules for the tokens you don't ever want to be split up by other rules, first. Put a string matcher before a word matcher.

That's still problematic as a solution for possibly overlapping rules, such as quoted strings and comments.