I was in a conversation the other day where Larry said that the reason he did that was to be able to say that Perl 5 had a smaller parser than did Perl 4. Now that he's refactoring the tokenizer, I think he regrets that choice somewhat.