...More thoughts...

There is a new edition of the book by Grune et al Parsing Techniques. It should be out or close.

Often people think about LR or LL parsing but the first edition of "Parsing Techniques" talks about starting anywhere! which is interesting at least in the sense of having maybe two-staged tokenizing <chess>?!</chess>

then there is HOP::Lexer. I haven't played much but it is quite slower than anything based on Parse::Yapp. Still a mixed approach could be nice...

finally using 5.9.x or more one can plug a different regexp engine, say a small one that permits "streaming"

Update: [SDF] seems also interesting.

cheers --stephan