The problem is that the standard Benchmark module doesn't isolate its test cases from one another. This means that the order that cases are run in can influence the results because side effects, either obvious or obscure, can accumulate and affect later tests.

Data in global variables is an obvious source of side effects; in the below example, the grep takes longer as more items are pushed onto the array, so the test functions that run later will be reported by Benchmark as being slower:

  cmpthese( 1000, {
    "test_1" => sub { push @global, scalar grep 1, @global },
    "test_2" => sub { push @global, scalar grep 1, @global },
    "test_3" => sub { push @global, scalar grep 1, @global },
  } );
[download]

To address this, I created a module that overrides the normal behavior of Benchmark to run each piece of code to be timed in a separate forked process. Just use Benchmark::Forking and the above benchmark reports the "correct" conclusion that the three tests run at approximately the same speed.

Feedback would be very welcome.

• Are there problems with the accuracy of this technique?
• Can you think of other examples of benchmark scripts this would not work for?
• Can anyone see a way to make this work like a subclass of the core Benchmark module rather than meddling with its internals?
• Is there something equivalent to this already in circulation?
• Any other issues I should consider before posting it to CPAN?

package Benchmark::Forking;

use strict;
use Benchmark;

use vars qw( $VERSION $Enabled $RunLoop );
BEGIN { 
  $VERSION = 0.9;
  $Enabled = 1; 
  $RunLoop = \&Benchmark::runloop;
}

sub enable   { $Enabled = 1 }
sub disable  { $Enabled = 0 }
sub enabled  { $#_ ? $Enabled = $_[1] : $Enabled }

sub import   { enable(); goto &Benchmark::import }
sub unimport { disable() }

sub Benchmark::runloop {
  $Enabled or return &$RunLoop;
  
  open( FORK, '-|') or print join "\n", @{ &$RunLoop } and exit;
  my @td = <FORK>;
  close( FORK ) or die $!;
  bless \@td, 'Benchmark';
};

1;
[download]

Update: I removed the old POD from this post, but it's still available here. An updated version is now on CPAN.)