I was swayed by reports I'd seen from people who swore that P::RD was the cat's meow. You're absolutely right that I gained insight from working with P::RD. Seeing those trace stacks going endlessly through the same fetches and recalculations was mortifying, not edifying! The key to a true RD algorithm is that your early results are saved in the recursive context as you call it again to go deeper. As you come back up the call stack, those results are available.

As for the tokenizer, I actually did a two-pass system, tokenizing the whole string and then working with tokens saved as TTYPE TVALUE. It would have been faster if I had only done one-token lookahead, but the maintenance advantages of separating the tokenizer from the solver outweighed the need for speed. I suspect that the tokenizer is far less a part of the problem than the recalcs. Even with shortcuts, not having access to the actual tokenset stack to replace sub-strings meant that everything was re-fetched again and again.

On your third point, it's a tribute to the languages we have now, especially Perl, that one CAN do so much with the base language. I'm soooo glad to have the choice, and to have the Open Source OSen available to run them, AND to have these multi-gigahertz PCs that labor so under Doze that just fly when set free with open source!!!