Lexical scoping is a very good thing because it means that when you use a module written by someone else, or when more than three or four people are working on a project at a time, a variable in one place won't clobber a variable somewhere else with the same name.

I won't say that the innards aren't a little messy (and I haven't read the regex code, so I don't have to roll for Sanity), but the way it works makes sense, and fits the Do What You Mean principle fairly well.

As for why my isn't the default in subroutines, there are two good reasons. First, it would have broken backwards compatibility between Perl 4 and Perl 5. The goal there was to minimize breakage. Second, it's the same reason 'use strict' isn't always on by default -- sometimes, quick and dirty scripts can use global variables and no one cares.

Through Perl 4 Perl had neither lexical scope nor real references. It's scoping mechanisms were package namespaces and dynamic scope through local.

With Perl 5 both of those were added. But they needed backwards compatibility. Therefore they could not by default localize variables in subs, nor could they change local's behaviour. And since the working implementation all used the symbol table (and therefore gave the wrong semantics), a lot of reimplementation and fixing needed to happen. Plus some thought needed to be given to the semantic issues of having so many different (and incompatible) kinds of scope.

Which means that over the 5.* series you have seen constant extension of what can be lexical and how much it is used. For instance from 5.003 to 5.004 they added lexical loop variables, and 5.6 adds lexically scoped declarations of access to global variables.

Remember that Perl is still at its heart a hack written to avoid writing a real report generator that has gone seriously astray. :-)