How do you pick this stuff up without delving into the internals?

That's how most of the important lexers/parsers work: they find the next token in a greedy way. That's why in most languages white space is only needed to separate tokens if concatenating them causes a different (and unwanted) way of parsing.

It's nothing specific to Perl, nor do you need to know anything about the internals.