Now, I'm about as far from expert on Perl's guts as possible, but I can see two possibilities:

1. It takes more time for Perl to backtrack for underscore (_) to $path than it takes to make the query (not impossible)
2. The version of Perl you're using doesn't optimize the different -X queries against the same file, and has to repeat the query each time. It takes some time to connect the "_" with the explicit filename.

I suspect the first hypothesis is more likely. After all, getting information about files is an extremely common operation in any O/S, and I would think it would be one which the O/S programmers would try very hard to make very fast.

Or, of course, I'm completely off the mark and you should listen to Fletch.