Have you considered checking hte return value of usleep (or sleep) to see how many actual seconds the method slept for, and then compare that with the delta from Time::HiRes::gettimeofday

I suspect that Time::HiRes::gettimeofday is in fact returning the accurate time for your system at the moment you call it, but you are seeing the larger then expected deltas because other things are happening on your system (besides the sleep) in between your calls -- Perl garbage collection, other processes on the same machine, etc....

My first suggestion was to try something like...

 time perl -MTime::HiRes -le 'print ((Time::HiRes::gettimeofday())[1])
+ for (1..500)'
[download]

...where you sanity check the numbers printed out looking for any really odd jumps, and Compare the "real" time reported by your system with the delta between the first/last times printed so see if they concur.

But the compilation/interpreter startup is significant enough to provide missleading info at such a small level of granularity.