You should probably take a look at Parse::Lex....

Also writing a small lexer in perl can be fairly trivial depending on your requirements (do you want to actually remove the items from the input stream or do you want to leave them, support regular expressions or not,etc...) Damian Conway actually gives an example of building an object oriented lexer in his book Object Oriented Perl.

cephas