Take a look at the article Lexing Your Data by Ovid on perl.com, in which he uses HOP::Lexer to tokenize SQL statements.

He did use a bit of a weird grammar, as a trick to circumvent some problems with that module, as I described in the thread HOP::Lexer not doing what I expected.

But otherwise, it should fit your bill perfectly.