Sure. As you can see, it's not the lexically first test that gets the biased. It's the first iteration of that test. Which explains why the bias is more pronounced the less runs you do.

By running all the tests once and discarding the results, you even up the playing field and the seconds cmpthese shows a much better distribution.

You should also consider shutting down as much else that is running on your box for the duration of the tests. For example, if my dial connection times out during a test, a high priority thread runs for the duration of the reconnect. That can completely skew the results.

Even using the mouse to pop up the task manager will have some effect. But if this is enough to obscure the gains you have made, it probably means that they are so small as to be subject to random variation anyway.

#! perl -slw
use strict;
use Benchmark qw[ cmpthese ];

our $ITERS ||= 5;
our $REPS  ||= 10000;

sub test { 
    my @strings = map{ ' ' x 1000 } 1 .. $REPS; 
}

my %tests = ( Atest => \&test, Btest => \&test, Ctest => \&test, Dtest
+ => \&test, );

## Ignore the results produced by this run
cmpthese( 1,      \%tests);

## These should show more even distribution. 
cmpthese( $ITERS, \%tests); 

P:\test>373536-2 -ITERS=10
        Rate Dtest Btest Ctest Atest
Dtest 4.27/s    --   -0%   -7%  -67%
Btest 4.27/s    0%    --   -7%  -67%
Ctest 4.59/s    7%    7%    --  -64%
Atest 12.8/s  200%  200%  179%    --
        Rate Ctest Dtest Atest Btest
Ctest 4.10/s    --   -1%   -1%   -1%
Dtest 4.13/s    1%    --   -0%   -0%
Atest 4.13/s    1%    0%    --   -0%
Btest 4.13/s    1%    0%    0%    --
[download]