In addition to profiling from the perl side, consider using strace -t to get a timestamped picture of what system calls are being made when and see if any latencies are readily apparent there (watch open() and stat() calls in particular).