Re: Wasting time thinking about wasted time

One thing you may have taken into account (though I probably missed) is that most filesystems, even network ones, tend to cache file attributes. The very first test that runs tends to get poor results. Even if the cache is warmed up, the caching behavior also changes as soon as you test on a large enough population of files. The cache in this case can turn into a kind of FIFO buffer, though you will still see caching effects (esp with sorts).

One way to help mitigate this is to test on a network filesystem with attribute caching disabled (e.g. many NFSs use the -noac option on the mount). You will probably still see server side caching effects, though I imagine the network delays will tend to dominate.

Another way to handle this might be to put the results for file attributes into a hash before the actual benchmark timing, and then introduce a constant delay during each lookup. (You could time how long it took to fill the initial hash and then use that to compute the delay, but then again this too can be affected by a warm cache...)

Comment on Re: Wasting time thinking about wasted time

Replies are listed 'Best First'.
Re^2: Wasting time thinking about wasted time by erikharrison (Deacon) on Sep 24, 2004 at 01:30 UTC
Generally working on a mounted filesystem for test one, then unmounting, then remounting, before running test two, eliminates cache issues for most OSes I run these kind of benchmarks against my usb thumbdrive. Easy to mount, easy to unmount, and the drive's so fast that it tends to reduce or eliminate disk speed confusion in benchmarks Cheers, Erik Light a man a fire, he's warm for a day. Catch a man on fire, and he's warm for the rest of his life. - Terry Pratchet	[reply]
Re^3: Wasting time thinking about wasted time by bluto (Curate) on Sep 27, 2004 at 15:28 UTC
Generally working on a mounted filesystem for test one, then unmounting, then remounting, before running test two, eliminates cache issues for most OSes When benchmarking in perl, I'm assuming that the first test case to run will incur the cold cache issues on it's very first run. After that every other test should run in the cache. It seems like you'd have to unmount every single test run -- not necessarily pleasant. It's Monday so I'm probably missing something...	[reply]