simonm: As a suggestion, it might be feasible to patch Benchmark.pm to get around this issue. ... You could fork and have each child only time one of the subroutines and then pass the data back to the parent for integration.

Here is a hack which implement's simonm's suggestion, which was also made independently to me by Gary Benson of Perl Seminar New York over a fine Indian meal at Angon on East 6 Street in Manhattan.

The hack involves three separate files and is probably modularizable, at least in part. To add/modify subroutines to be tested, add them to the third file below.

##### FILE 1 of 3: bmharness.pl ##### #!/usr/local/bin/perl use strict; use warnings; use Benchmark qw( timethis ); # bmharness.pl # Perl script which spawns processes in which &Benchmark::timethis # times subroutines, 1 subroutine per process. # Output is then collected and returned to parent. # Objective: Avoid biases in results calculated by # &Benchmark::timethis and &Benchmark::timethese when comparing # two or more subroutines involving heavy # computation within a single process. # Usage: bmharness.pl codefile [iterations] subroutines... # codefile: a file which contains: # (a) all subroutines to be tested, # (b) all modules which must be loaded for those subs to run and # (c) any code needed prior to calling those subs # iterations: # defaults to 10 per sub unless otherwise stated on command-line # subroutines: the particular subroutines defined in codefile # which you wish to test with a given invocation of bmharness.pl my ($iterations, @subs, $codefile, @outputs); die "No command-line arguments provided: $!" unless @ARGV; $codefile = shift(@ARGV); die "Cannot locate file $codefile containing subs to be tested: $!" unless (-f $codefile); $iterations = ($ARGV[0] =~ /^\d+$/ and $ARGV[0] > 0) ? shift(@ARGV) : 10; @subs = @ARGV; foreach my $sub (@subs) { my $out = qx{ perl bmharnessengine.pl $codefile $iterations $sub }; push(@outputs, $out); } print "$_\n" for @outputs; print "\nFinished\n"; ##### FILE 2 of 3: bmharnessengine.pl ##### #!/usr/local/bin/perl use strict; use warnings; use Benchmark qw( timethis ); # bmharnessengine.pl # Perl script used internally within bmharness.pl; # do not call directly my ($codefile, $iterations, $sub); die "No command-line arguments provided: $!" unless @ARGV; $codefile = shift(@ARGV); require $codefile; $iterations = shift(@ARGV); $sub = shift(@ARGV); benchmark_harness($iterations, $sub); sub benchmark_harness { my $iterations = shift; my $sub = shift; die "Sub $sub not defined in codefile $codefile: $!" unless defined &{$sub}; print "Testing $iterations iterations of sub $sub ...\n"; timethis($iterations, \&{$sub}); } ##### FILE 3 of 3: libsubtesting ##### # libsubtesting # Sample of a library file in which you would place subroutines # you wish to test via bmharness.pl # Illustration: # To compare List::Compare and List::Compare::Functional # with respect to getting # five different set relationships, testing each for 100 iterations, # call: bmharness.pl libsubtesting 100 getsets_lc getsets_lcf use List::Compare; use List::Compare::Functional qw( get_intersection get_union get_unique get_symdiff get_complement ) +; my (@listrefs); @listrefs = ( [ 1..3000 ], [ 2251..5000 ], [ 2751..3500 ], ); print 'Testing ' . @listrefs . ' lists of '; my ($sizestr); foreach my $list (@listrefs) { $sizestr .= scalar(@{$list} . ' '); } print "$sizestr\n\n"; my (@intersection, @union, @unique, @complement, @symdiff); ##### SUBROUTINES ##### sub getsets_lc { my $lc = List::Compare->new(@listrefs); @intersection = $lc->get_intersection(); @union = $lc->get_union(); @unique = $lc->get_unique(2); @complement = $lc->get_complement(2); @symdiff = $lc->get_symdiff(); } sub getsets_lcf { @intersection = get_intersection( [ @listrefs ] ); @union = get_union( [ @listrefs ] ); @unique = get_unique( [ @listrefs ], [2] ); @complement = get_complement( [ @listrefs ], [2] ); @symdiff = get_symdiff( [ @listrefs ] ); } 1;


In reply to Re^2: Benchmark.pm: Does subroutine testing order bias results? by jkeenan1
in thread Benchmark.pm: Does subroutine testing order bias results? by jkeenan1

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.