There's a feature with stat() where it caches the item being tested on first call, to reduce overhead that would likely help reduce checking time. Note the _ at the end of the one-liner below. That's the cached item.

perl -wMstrict -E 'say -f "utf.pl" && say -e _'
[download]

Also, one could also look at the no_chdir => 1 feature of File::Find, however I don't know if one would get different warnings in that case, or if it'd just fail silently and move along when it encountered a restricted dir.