in reply to Re: Benchmark.pm: Does subroutine testing order bias results?
in thread Benchmark.pm: Does subroutine testing order bias results?

simonm: As a suggestion, it might be feasible to patch Benchmark.pm to get around this issue. ... You could fork and have each child only time one of the subroutines and then pass the data back to the parent for integration.

Here is a hack which implement's simonm's suggestion, which was also made independently to me by Gary Benson of Perl Seminar New York over a fine Indian meal at Angon on East 6 Street in Manhattan.

The hack involves three separate files and is probably modularizable, at least in part. To add/modify subroutines to be tested, add them to the third file below.

##### FILE 1 of 3: bmharness.pl ##### #!/usr/local/bin/perl use strict; use warnings; use Benchmark qw( timethis ); # bmharness.pl # Perl script which spawns processes in which &Benchmark::timethis # times subroutines, 1 subroutine per process. # Output is then collected and returned to parent. # Objective: Avoid biases in results calculated by # &Benchmark::timethis and &Benchmark::timethese when comparing # two or more subroutines involving heavy # computation within a single process. # Usage: bmharness.pl codefile [iterations] subroutines... # codefile: a file which contains: # (a) all subroutines to be tested, # (b) all modules which must be loaded for those subs to run and # (c) any code needed prior to calling those subs # iterations: # defaults to 10 per sub unless otherwise stated on command-line # subroutines: the particular subroutines defined in codefile # which you wish to test with a given invocation of bmharness.pl my ($iterations, @subs, $codefile, @outputs); die "No command-line arguments provided: $!" unless @ARGV; $codefile = shift(@ARGV); die "Cannot locate file $codefile containing subs to be tested: $!" unless (-f $codefile); $iterations = ($ARGV[0] =~ /^\d+$/ and $ARGV[0] > 0) ? shift(@ARGV) : 10; @subs = @ARGV; foreach my $sub (@subs) { my $out = qx{ perl bmharnessengine.pl $codefile $iterations $sub }; push(@outputs, $out); } print "$_\n" for @outputs; print "\nFinished\n"; ##### FILE 2 of 3: bmharnessengine.pl ##### #!/usr/local/bin/perl use strict; use warnings; use Benchmark qw( timethis ); # bmharnessengine.pl # Perl script used internally within bmharness.pl; # do not call directly my ($codefile, $iterations, $sub); die "No command-line arguments provided: $!" unless @ARGV; $codefile = shift(@ARGV); require $codefile; $iterations = shift(@ARGV); $sub = shift(@ARGV); benchmark_harness($iterations, $sub); sub benchmark_harness { my $iterations = shift; my $sub = shift; die "Sub $sub not defined in codefile $codefile: $!" unless defined &{$sub}; print "Testing $iterations iterations of sub $sub ...\n"; timethis($iterations, \&{$sub}); } ##### FILE 3 of 3: libsubtesting ##### # libsubtesting # Sample of a library file in which you would place subroutines # you wish to test via bmharness.pl # Illustration: # To compare List::Compare and List::Compare::Functional # with respect to getting # five different set relationships, testing each for 100 iterations, # call: bmharness.pl libsubtesting 100 getsets_lc getsets_lcf use List::Compare; use List::Compare::Functional qw( get_intersection get_union get_unique get_symdiff get_complement ) +; my (@listrefs); @listrefs = ( [ 1..3000 ], [ 2251..5000 ], [ 2751..3500 ], ); print 'Testing ' . @listrefs . ' lists of '; my ($sizestr); foreach my $list (@listrefs) { $sizestr .= scalar(@{$list} . ' '); } print "$sizestr\n\n"; my (@intersection, @union, @unique, @complement, @symdiff); ##### SUBROUTINES ##### sub getsets_lc { my $lc = List::Compare->new(@listrefs); @intersection = $lc->get_intersection(); @union = $lc->get_union(); @unique = $lc->get_unique(2); @complement = $lc->get_complement(2); @symdiff = $lc->get_symdiff(); } sub getsets_lcf { @intersection = get_intersection( [ @listrefs ] ); @union = get_union( [ @listrefs ] ); @unique = get_unique( [ @listrefs ], [2] ); @complement = get_complement( [ @listrefs ], [2] ); @symdiff = get_symdiff( [ @listrefs ] ); } 1;