"...I got bitten by the fact that globbing is a shell operation"

This is true only in versions of perl prior to 5.6. In 5.6, globbing is implemented internally using File::Glob. My benchmarking shows glob() in 5.6 to be two orders of magnitude faster than in prior versions. Not only that, but it is also faster than the opendir(), readdir(), grep idiom. (yes, I am aware of your objections to 5.6, but I have yet to encounter any bugs, personally)