Never use q{...} with Benchmark; use sub {...} instead. It's too easy to make a scope mistake when using uncompiled code.
Hmm. Mistakes are mistakes. Using our for "setup variables" used within the tests but declared external to it, is a simple solution to the scoping problem.
Whilst using coderefs is an effective ways of capturing external variables through closure, it has other side effects that can influence your benchmarks in much more subtle and harder to detect ways. This is especially true for the often seen 'communal' benchmark where different monks implementations of a subroutine are tested and there are parameters to the subs:
sub monk1 {
my( $arg, @data ) = @_;
## do stuff
return @results;
}
sub monk2 {
my( $arg, @data ) = @_;
## do stuff
return @results;
}
....
my @data = (....);
cmpthese -3, {
monk1 => sub{ my @results = monk1( @data ); },
monk2 => sub{ my @results = monk2( @data ); },
....
};
The problem here is that you are adding an extra level of subroutine call to each of the implementations under test. If the work being done inside the subroutine is substantial, then the extra overhead will be lost in the noise of the testing and have no substantial effect upon the outcome. But if the code being tested is itself not doing a great deal, then the additional overhead of the extra level of sub call can swamp the differences in the implementation and render the benchmark useless as a result.
To demonstrate this effect using the grep example, if we grep a list of 10 items using your prefered coderef form of benchmark, we get these results:
10 items
Rate sub_block_short sub_list_short
sub_block_short 305975/s -- -4%
sub_list_short 319968/s 5% --
from which you might conclude that the list form of grep is 5% faster than the block form. However, if you perform the same test using the q[ ... ] form of benchmark you get:
10 items
Rate eval_block_short eval_list_short
eval_block_short 322569/s -- -1%
eval_list_short 325028/s 1% --
From which you might conclude that the list form is only 1% quicker! Assuming an otherwise quiescent system, this is a rather less dramatic difference, that is probably confined to the realms of 'noise', and definitly not worth bothering with if you consider the block form clearer than the list form.
And the reason for the difference between the two tests is the overhead of the extra layer of subroutine call in the coderef form of the benchmark. That extra overhead completely obscures the actual differences we are attempting to assertain and explore. In this case, using the block form renders the benchmark useless as the construction of the benchmark itself has influenced and obscured the results we are attempting to define.
This stands out more when you campare the 4 forms together:
10 items
Rate sub_blk_sh sub_lst_sh eval_blk_sh eval_lst_
+sh
sub_blk_sh 305975/s -- -4% -5% -
+6%
sub_lst_sh 319968/s 5% -- -1% -
+2%
eval_blk_sh 322569/s 5% 1% -- -
+1%
eval_lst_sh 325028/s 6% 2% 1%
+--
With the eval_block form coming out as 5% faster to the sub_list form despite that the block form has to be slower due to the construction of the extra level of scope!
That clearly demonstrates that the construction of the benchmark itself can have a fairly dramatic influence upon the results obtained.
Of course, as the number of items in the list increases, so the effect of the extra overhead becomes ammortised over an increasing large number of scopes generations and has much less influence on the outcome.
100_000 items
Rate eval_blk_vl sub_blk_vl eval_lst_vl sub_lst_vl
eval_blk_vl 36.4/s -- -1% -3% -4%
sub_blk_vl 36.6/s 1% -- -2% -3%
eval_lst_vl 37.4/s 3%* 2% -- -1%
sub_list_vl 37.8/s 4% 3%* 1% --
Here we see that the differences between the eval_list & block forms, and those between the sub_list & block forms have equalled out at 3%. Less dramatic than the 5% and more significant than 1%.
However, you will also notice that here the sub forms have overtaken the eval forms. What happened to that extra overhead? If there is an extra overhead attached to the coderef form of benchmark, you would expect that the eval forms would remain faster than their equivalent coderef forms. The differences would be reduced as the overhead is amortised, but still the overhead should be present and show up.
The conclusion I draw from that is that, at these levels, the cost of constructing the extra scope required by the block form of grep is so insignificant that it simply gets lost in the noise of the benchmark; the internal inconsistancies of the Benchmark module itself; the impact of Perl's memory management; and external factors relating to system usage. As such, I long ago abandoned the list form of grep in favour of the (IMO), syntactically clearer block form as the efficiency justification of using the list form simply doesn't hold up in use.
Even for the most highly iterated, deeply nested uses of grep, the performance of the construct is much more likely to be influenced by whether I hit a graphic heavy page in my browser; or list the files in my directory; or even just wiggle my mouse about a little; than whether I have used the list or block form.
If there are any overall conclusions to be drawn, they are:
- Benchmarking well is hard.
- Drawing the right conclusions from any given benchmark is even harder.
- And never, ever say(*) "Never use ..." or "Always use ..." or "You should (not) do ...".
(*)Except in this case of course :)
Benchmark code and full results:
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
|