Of course, the biggest flaw is the use the Benchmark module itself. How can use trust a module that sometimes reports negative times?

I do remember seeing that a long time ago, I've not encountered it with recent versions?

However, the main reason for using it is that it seems to get the math right more often than not. Whereas looking at your math above, I am pretty certain that it is suspect.

• Your iteration count is 10_000_000, but your divisor is 1_000_000.
• Without having analysed it too closely, you appear to have a precedence problem in your delta calculations.

  This

  my $d2 = $s3 - $s2 + ($m3 - $m2) / 1_000_000;
  [download]

  is parsed as

  (my $d2 = (($s3 - $s2) + (($m3 - $m2) / 1000000)));
  [download]

  which means that you are adding 1 millionth of the difference in the milliseconds to the difference in the seconds.

  Which is about as close to a random number as I can think of :)

Once you fix those problems up, your benchmark method produces much the same results as mine above. And does so consistantly:

c:\test>junk2 -ITERS=1e6
And: 0.000000300  Mod: 0.000000372

c:\test>junk2 -ITERS=10e6
And: 0.000000302  Mod: 0.000000336

c:\test>junk2 -ITERS=100e6
And: 0.000000299  Mod: 0.000000338
[download]

You got me on the one counter issue, though if it makes a difference, perl's math is really bad.

But then your use of the global $a is pretty suspect also. Globals are slower than lexicals, so using them has a significant affect upon the results, but switching to lexicals muddies the waters also:

c:\test>junk2 -ITERS=1e6
And: 0.000000403  Mod: 0.000000347

c:\test>junk2 -ITERS=1e6
And: 0.000000409  Mod: 0.000000325

c:\test>junk2 -ITERS=10e6
And: 0.000000386  Mod: 0.000000339

c:\test>junk2 -ITERS=10e6
And: 0.000000384  Mod: 0.000000338
[download]

Besides benchmarking slower, it switches the balance of the performance of the operations! The slowdown may be to do with allocation/deallocation of lexical variables(?), but why the type of variable you assign the result of teh expression to should have such a profound significance is beyond me?

That's why I dodged the issue altogether and used ++$counter <op> and 1;. I added this change, to fixes for the problems described above to produce these results:

c:\test>junk2 -ITERS=1e6
And: 0.000000277  Mod: 0.000000286

c:\test>junk2 -ITERS=1e6
And: 0.000000276  Mod: 0.000000271

c:\test>junk2 -ITERS=1e6
And: 0.000000248  Mod: 0.000000283

c:\test>junk2 -ITERS=10e6
And: 0.000000252  Mod: 0.000000277

c:\test>junk2 -ITERS=10e6
And: 0.000000252  Mod: 0.000000278

c:\test>junk2 -ITERS=10e6
And: 0.000000255  Mod: 0.000000278

c:\test>junk2 -ITERS=100e6
And: 0.000000250  Mod: 0.000000277
[download]

Which are a) entirely self consistant; b) are consistant with the results produced by two other benchmarking methods--both my use of Benchmark and lidden's use of an external timer.

The corrected benchmark is:

#! perl -slw
use strict;
use Time::HiRes 'gettimeofday';

our $ITERS ||= 10_000_000;

my $counter1  = 0;
my $counter2  = 0;

my $s1 = gettimeofday;
for (1 .. $ITERS) { ++$counter1 & 1 and 1 }
my $s2 = gettimeofday;
for (1 .. $ITERS) { ++$counter2 % 2 and 1 }
my $s3 = gettimeofday;

my $d1 = ( $s2 - $s1 ) / $ITERS;
my $d2 = ( $s3 - $s2 ) / $ITERS;

printf "And: %.9f  Mod: %.9f\n", $d1, $d2;
[download]

Your iteration count is 10_000_000, but your divisor is 1_000_000

Indeed. Why should they be the same? The interator counts iterations. The divisor divides the number of microseconds. I'd divide by a million regardless how many times I iterated.

This

my $d2 = $s3 - $s2 + ($m3 - $m2) / 1_000_000;
[download]

is parsed as

(my $d2 = (($s3 - $s2) + (($m3 - $m2) / 1000000)));
[download]

Indeed it is! Exactly as I intent it! Surprise, surprise.

which means that you are adding 1 millionth of the difference in the milliseconds to the difference in the seconds.

Wrong. RTFM. gettimeofday returns the number of microseconds. And guess what? There are a million microseconds in a second. Even in the USA and the UK.

Now, beside being wrong, you are also inconsistent. In your first point, you accuse me of not dividing by the number of iterations, as if the value wouldn't be a number of sub-second units. Then in your second point you do think it's a number of sub-second units. You can't have it both ways.

You're right. I never use gettimeofday, and mistakenly assumed it returned millseconds not microseconds.

Now, beside being wrong, you are also inconsistent. In your first point, you accuse me of not dividing by the number of iterations, as if the value wouldn't be a number of sub-second units. Then in your second point you do think it's a number of sub-second units. You can't have it both ways.

I misunderstood your arcane way of doing floating point math and thought you were dividing by the (wrong) number of iterations. I misundertsood your code, for which I apologise, but there is no inconsistancy. Good enough reason to use a module rather than hand code the math.

I also noted "Without having analysed it too closely, you appear to have a precedence problem in your delta calculations.".

That said, even if we return to using your arcane floating point math and bizarre failure to break down the results on a per iteration basis, the results of your benchmark method, modified by the removal of your $a fopar, still shows that & 1 is consistantly faster than % 2.

#! perl -slw
use strict;
use Time::HiRes 'gettimeofday';

our $ITERS ||= 10_000_000;

my $counter1  = 0;
my $counter2  = 0;

my( $s1, $m1 ) = gettimeofday;
for (1 .. $ITERS) { ++$counter1 & 1 and 1 }
my( $s2, $m2 ) = gettimeofday;
for (1 .. $ITERS) { ++$counter2 % 2 and 1 }
my( $s3, $m3 ) = gettimeofday;

my $d1 = $s2 - $s1 + ($m2 - $m1) / 1_000_000;
my $d2 = $s3 - $s2 + ($m3 - $m2) / 1_000_000;

printf "And: %.9f  Mod: %.9f\n", $d1, $d2;

__END__
c:\test>junk2 -ITERS=1e6
And: 0.258308000  Mod: 0.304192000

c:\test>junk2 -ITERS=1e6
And: 0.258755000  Mod: 0.272495000

c:\test>junk2 -ITERS=1e6
And: 0.259628000  Mod: 0.302872000

c:\test>junk2 -ITERS=10e6
And: 2.656250000  Mod: 2.828125000

c:\test>junk2 -ITERS=10e6
And: 2.671875000  Mod: 2.859375000

c:\test>junk2 -ITERS=10e6
And: 2.671875000  Mod: 2.859375000

c:\test>junk2 -ITERS=100e6
And: 26.187500000  Mod: 28.375000000

c:\test>junk2 -ITERS=100e6
And: 26.265625000  Mod: 28.328125000
[download]

And the more iterations you run (therebye reducing the obscuring affects of the invariant parts of the code under test), the more consistant it becomes.

Just as was shown by both my Benchmark method and lidden's external timer method.

---

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

The reason the code assigns to a variable outside the scope of block is to avoid a smart optimizer optimizing the '&' or '% 2' away, as the result isn't used at all. I didn't want to dive into the Perl source code and see whether '&' or '%' checked their context and decide to do nothing in void context. If one or both would do so, the benchmark would be flawed. It's the same for an assignment to a variable that's about to go out of scope; if the variable doesn't have magic attached, one might not do the assignment after all. I've been bitten by the optimizations Perl makes so many times when benchmarking, I'm very careful and try to give the leave as little room for short cuts as possible. Hence my preference for assigning to a variable whose scope goes beyond the block being tested.

As for your version, Perl parses ++$counter1 & 1 and 1 as '???' if ++$counter1 & 1. I don't know how much that means for the benchmark.

FYI: From the Deparse POD:

Expand conventional syntax constructions into equivalent ones that expose their internal operation. LEVEL should be a digit, with higher values meaning more expansion. As with -q, this actually involves turning off special cases in B::Deparse's normal operations.

c:\test>perl -MO=Deparse,-x9 -e"++$counter &1 and 1"
++$counter & 1 and '???';
-e syntax OK
[download]

And to convince you that actually does what is required:

c:\test>perl -e"++$counter &1 and 1; print $counter"
1
[download]

---

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

I do remember seeing that a long time ago, I've not encountered it with recent versions?

The thing causing this behaviour is still there, running empty loops and subtracting times.

Here's an example:

#!/usr/bin/perl

use strict;
use warnings;

use Benchmark;

timethese -1, {
    trivial1  => sub {1},
    trivial2  => sub {2},
};

__END__

Benchmark: running trivial1, trivial2 for at least 1 CPU seconds...
  trivial1:  1 wallclock secs ( 1.13 usr +  0.00 sys =  1.13 CPU) @ 44
+79726.55/s (n=5062091)
  trivial2:  3 wallclock secs ( 1.97 usr + -0.01 sys =  1.96 CPU) @ 43
+171185.71/s (n=84615524)
[download]

BTW, it took about 5 _minutes_ to run this benchmark. And for some reason, it decided to run 'trivial2' 16 times as many times as 'trivial1'.

What version of perl are you using?

---
$world=~s/war/peace/g

And for some reason, it decided to run 'trivial2' 16 times as many times as 'trivial1'.

The reason is the same as the problem with the OP's original benchmark.

c:\test>perl -mstrict -we"1;"

c:\test>perl -mstrict -we"2;"
Useless use of a constant in void context at -e line 1.
[download]

1 is a special cases for Perl constants that does not get optimised away, and is the reason why I used  ... and 1; to avoid the "Useless use in a void context" in my benchmark.

This special case is so that things like these work:

while( 1 ) {...
... if 1;
1 while ....;
[download]

However, 2 is not special and so gets optimised away. Hence, trivial1 takes longer than trivial2, so the loop has to be run many, many times more in order to accumulate the "for at least 1 second of cpu" in

timethese -1, {
    trivial1  => sub {1},
    trivial2  => sub {2},
};;
Benchmark:
running
  trivial1, trivial2
 for at least 1 CPU seconds
...
[Range iterator outside integer range at (eval 57) line 1, <STDIN> lin
+e 7.
[download]

I guess my machine is faster than yours. Faster enough that when Benchmark attempted to run the loop for sufficient iterations to accumulate the required cpu usage, it encountered my pet hate of the perl iterator!

it took about 5 _minutes_ to run this benchmark.

Unsurprising. When the bodies of the iterator loops is doing next to nothing, or actually nothing, when Benchmark does it initial timings of them in order to calculate the number of iterations to run it for, it attempts to subtract a small amont to account for the overhead of the loop itself, with the result that the calculation are probably being subjected to rounding errors.

When it takes 84 million iterations of a test to accumulate 1 second of cpu on a modern processor, it certainly indicates that something is wrong with your benchmark.

This is why I tend to incorprate for loops within the test when benchmarking very small pieces of code, rather than relying on the benchmark iteration count.

---

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.