in reply to Re^4: Removing Flanking "N"s in a DNA String
in thread Removing Flanking "N"s in a DNA String

There are two problems with your benchmark.

  1. The time taken to copy the data to modify within the code being benchmarked, drowns the minscule time spent do so.
  2. Your benchmark does not account for Benchmark.pms habit of preferencial biasing the tests in favour of the first case run.

    In the following, the only change I have made to your benchmark is to reverse the naming of the cases so tha they will be run in teh reverse order. Note how in most cases the "winner" is reversed, and in the few where this not the case, the differences are within the bounds of experimental error:

    #!/usr/bin/perl use strict; use warnings; use Benchmark qw( cmpthese ); sub run_tests { my ( $len_remove, $len_keep, $num_repeat ) = @_; my $remove = 'N' x $len_remove; my $keep = 'O' x $len_keep; my %test = ( front => "$remove$keep" x $num_repeat, tail => "$keep$remove" x $num_repeat, both_ends => "$remove$keep" x $num_repeat . $remove, nothing => "$keep$remove" x $num_repeat . $keep, ); print "$len_remove chars to remove, $len_keep chars long kept sequ +ences, $num_repeat repetitions.\n"; for my $type ( keys %test ) { print "Measuring removing at $type.\n"; cmpthese -2 => { two_sub => sub { for( 1 .. 1000 ) { s{^N*(.*?)N*$}{$1} for + my $copy = $test{$type} } }, one_sub => sub { for( 1 .. 1000 ) { s{^N*}{}, s{N*$}{} for + my $copy = $test{$type} } }, }; } print "\n"; } $|++; run_tests 4, 4, 1; run_tests 20, 20, 1; run_tests 20, 20, 50; run_tests 4, 4, 20; run_tests 4, 12, 10; run_tests 4, 100, 100; __END__ P:\test>junk3 4 chars to remove, 4 chars long kept sequences, 1 repetitions. Measuring removing at front. Rate two_sub one_sub two_sub 101/s -- -64% one_sub 279/s 177% -- Measuring removing at tail. Rate two_sub one_sub two_sub 103/s -- -64% one_sub 284/s 176% -- Measuring removing at nothing. Rate two_sub one_sub two_sub 98.5/s -- -54% one_sub 215/s 118% -- Measuring removing at both_ends. Rate two_sub one_sub two_sub 97.2/s -- -64% one_sub 269/s 177% -- 20 chars to remove, 20 chars long kept sequences, 1 repetitions. Measuring removing at front. Rate two_sub one_sub two_sub 81.8/s -- -49% one_sub 160/s 96% -- Measuring removing at tail. Rate two_sub one_sub two_sub 83.7/s -- -49% one_sub 164/s 96% -- Measuring removing at nothing. Rate two_sub one_sub two_sub 60.4/s -- -29% one_sub 85.0/s 41% -- Measuring removing at both_ends. Rate two_sub one_sub two_sub 77.6/s -- -55% one_sub 172/s 121% -- 20 chars to remove, 20 chars long kept sequences, 50 repetitions. Measuring removing at front. Rate one_sub two_sub one_sub 3.47/s -- -8% two_sub 3.77/s 8% -- Measuring removing at tail. Rate one_sub two_sub one_sub 3.44/s -- -7% two_sub 3.71/s 8% -- Measuring removing at nothing. Rate one_sub two_sub one_sub 3.47/s -- -6% two_sub 3.68/s 6% -- Measuring removing at both_ends. Rate one_sub two_sub one_sub 3.56/s -- -5% two_sub 3.74/s 5% -- 4 chars to remove, 4 chars long kept sequences, 20 repetitions. Measuring removing at front. Rate two_sub one_sub two_sub 34.1/s -- -17% one_sub 41.1/s 20% -- Measuring removing at tail. Rate two_sub one_sub two_sub 35.1/s -- -14% one_sub 41.0/s 17% -- Measuring removing at nothing. Rate two_sub one_sub two_sub 34.1/s -- -12% one_sub 38.9/s 14% -- Measuring removing at both_ends. Rate two_sub one_sub two_sub 33.6/s -- -18% one_sub 41.1/s 22% -- 4 chars to remove, 12 chars long kept sequences, 10 repetitions. Measuring removing at front. Rate two_sub one_sub two_sub 35.2/s -- -15% one_sub 41.2/s 17% -- Measuring removing at tail. Rate two_sub one_sub two_sub 35.2/s -- -16% one_sub 41.7/s 19% -- Measuring removing at nothing. Rate two_sub one_sub two_sub 30.6/s -- -18% one_sub 37.2/s 22% -- Measuring removing at both_ends. Rate two_sub one_sub two_sub 34.4/s -- -17% one_sub 41.5/s 21% -- 4 chars to remove, 100 chars long kept sequences, 100 repetitions. Measuring removing at front. (warning: too few iterations for a reliable count) (warning: too few iterations for a reliable count) s/iter one_sub two_sub one_sub 1.44 -- -10% two_sub 1.29 11% -- Measuring removing at tail. (warning: too few iterations for a reliable count) (warning: too few iterations for a reliable count) s/iter one_sub two_sub one_sub 1.43 -- -11% two_sub 1.27 12% -- Measuring removing at nothing. (warning: too few iterations for a reliable count) (warning: too few iterations for a reliable count) s/iter one_sub two_sub one_sub 1.45 -- -13% two_sub 1.27 15% -- Measuring removing at both_ends. (warning: too few iterations for a reliable count) (warning: too few iterations for a reliable count) s/iter one_sub two_sub one_sub 1.46 -- -13% two_sub 1.27 15% --

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

Replies are listed 'Best First'.
Re^6: Removing Flanking "N"s in a DNA String
by Perl Mouse (Chaplain) on Nov 07, 2005 at 16:35 UTC
    What do you mean "the winner is reversed"? Sure, the names are reversed, but that's to be expected as you reversed the names in the test to prove a point.

    Which I think, you didn't, as the test still favour the solution that uses two substitutions - just naming it one_sub doesn't change that.

    Perl --((8:>*

      You're right. Dumb conclusion.

      I still stand by my benchmark though--until someone shows me what is wrong with it.

      By moving the test data generation outside of the test, and testing each method twice in reversed order, I believe that I have accounted for the Benchmark.PM "first case" bias, and am more accurately comparing the two methods than any of the other benchmarks posted.

      Can you dubunk that claim?


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.