I don't know if you've been following Marpa::R2, but they've been going in the direction of putting tokenizer rules in the grammar, and it looks rather good.

No, I haven't. From my brief interactions with the author I had no expectation that Marpa would ever change in any way that would cause me to reconsider it.

From a cursory scan of the R2 docs it does seem that somewhere amongst the close to 2 dozen modules that make it up there might be something that starts to look like it might do the job. But, it is really hard to tell given that the two examples are:

• an expression parser that uses about the same number of source lines as I did for my 6 function/3 precedence level expression parser -- on top of those 2 dozens modules -- to provide a 2 function/1 precedence level expression parser.
• The other deals with one of those completely pointless, meaningless, "languages" that does absolutely nothing, that parser theorists are so enamored of.

If I ever find a working example of a block-structured language done using Marpa, I might look again; but I won't hold my breath.

Unfortunately, the documentation doesn't seem to have improved. It still spends an inordinate amount of time telling me haw clever the parser is; and almost none showing me how to use it to do something realistic and useful.