Of course, the biggest flaw is the use the Benchmark module itself. How can use trust a module that sometimes reports negative times?

I do remember seeing that a long time ago, I've not encountered it with recent versions?

However, the main reason for using it is that it seems to get the math right more often than not. Whereas looking at your math above, I am pretty certain that it is suspect.

• Your iteration count is 10_000_000, but your divisor is 1_000_000.
• Without having analysed it too closely, you appear to have a precedence problem in your delta calculations.

  This

  my $d2 = $s3 - $s2 + ($m3 - $m2) / 1_000_000;
  [download]

  is parsed as

  (my $d2 = (($s3 - $s2) + (($m3 - $m2) / 1000000)));
  [download]

  which means that you are adding 1 millionth of the difference in the milliseconds to the difference in the seconds.

  Which is about as close to a random number as I can think of :)

Once you fix those problems up, your benchmark method produces much the same results as mine above. And does so consistantly:

c:\test>junk2 -ITERS=1e6
And: 0.000000300  Mod: 0.000000372

c:\test>junk2 -ITERS=10e6
And: 0.000000302  Mod: 0.000000336

c:\test>junk2 -ITERS=100e6
And: 0.000000299  Mod: 0.000000338
[download]

You got me on the one counter issue, though if it makes a difference, perl's math is really bad.

But then your use of the global $a is pretty suspect also. Globals are slower than lexicals, so using them has a significant affect upon the results, but switching to lexicals muddies the waters also:

c:\test>junk2 -ITERS=1e6
And: 0.000000403  Mod: 0.000000347

c:\test>junk2 -ITERS=1e6
And: 0.000000409  Mod: 0.000000325

c:\test>junk2 -ITERS=10e6
And: 0.000000386  Mod: 0.000000339

c:\test>junk2 -ITERS=10e6
And: 0.000000384  Mod: 0.000000338
[download]

Besides benchmarking slower, it switches the balance of the performance of the operations! The slowdown may be to do with allocation/deallocation of lexical variables(?), but why the type of variable you assign the result of teh expression to should have such a profound significance is beyond me?

That's why I dodged the issue altogether and used ++$counter <op> and 1;. I added this change, to fixes for the problems described above to produce these results:

c:\test>junk2 -ITERS=1e6
And: 0.000000277  Mod: 0.000000286

c:\test>junk2 -ITERS=1e6
And: 0.000000276  Mod: 0.000000271

c:\test>junk2 -ITERS=1e6
And: 0.000000248  Mod: 0.000000283

c:\test>junk2 -ITERS=10e6
And: 0.000000252  Mod: 0.000000277

c:\test>junk2 -ITERS=10e6
And: 0.000000252  Mod: 0.000000278

c:\test>junk2 -ITERS=10e6
And: 0.000000255  Mod: 0.000000278

c:\test>junk2 -ITERS=100e6
And: 0.000000250  Mod: 0.000000277
[download]

Which are a) entirely self consistant; b) are consistant with the results produced by two other benchmarking methods--both my use of Benchmark and lidden's use of an external timer.

The corrected benchmark is:

#! perl -slw
use strict;
use Time::HiRes 'gettimeofday';

our $ITERS ||= 10_000_000;

my $counter1  = 0;
my $counter2  = 0;

my $s1 = gettimeofday;
for (1 .. $ITERS) { ++$counter1 & 1 and 1 }
my $s2 = gettimeofday;
for (1 .. $ITERS) { ++$counter2 % 2 and 1 }
my $s3 = gettimeofday;

my $d1 = ( $s2 - $s1 ) / $ITERS;
my $d2 = ( $s3 - $s2 ) / $ITERS;

printf "And: %.9f  Mod: %.9f\n", $d1, $d2;
[download]