Re: Initializing iterations while benchmarking

You're right to be concerned. These are the results of your orignal on my system:

C:\test>786728-1.pl
          Rate three   one   two
three  95432/s    --  -13%  -35%
one   110191/s   15%    --  -25%
two   146082/s   53%   33%    --
[download]

And these excluding the setup overhead:

C:\test>786728-2.pl
           Rate   one   two three
one   6935150/s    --   -0%   -6%
two   6965018/s    0%    --   -5%
three 7345874/s    6%    5%    --
[download]

As you can see, the setup swamps the code under test and skws the results horribly.

Here's my version of the benchmark:

use strict;
use warnings;
use Benchmark;

our @strings = qw(exception:tex exception:mex asdf tex:exception:mex);

Benchmark::cmpthese( -5, {
    'one'   => q[ my @filtered = grep { /exception:(?!tex)/} @unfilter
+ed; ],
    'two'   => q[ my @filtered = grep { /exception/ && !/tex/ } @unfil
+tered; ],
    'three' => q[ my @filtered = grep { /exception:/g && !/\Gtex/ } @u
+nfiltered; ],
});
[download]

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

RIP PCW

Comment on Re: Initializing iterations while benchmarking Select or Download Code

Replies are listed 'Best First'.
Re^2: Initializing iterations while benchmarking by ig (Vicar) on Aug 07, 2009 at 10:34 UTC
Thanks BrowserUk but in your example @unfiltered is empty. `use strict; use warnings; use Benchmark; our @strings = qw(exception:tex exception:mex asdf tex:exception:mex); Benchmark::cmpthese( -5, { 'one' => q[ my @filtered = grep { /exception:(?!tex)/} @strings; + ], 'two' => q[ my @filtered = grep { /exception/ && !/tex/ } @strin +gs; ], 'three' => q[ my @filtered = grep { /exception:/g && !/\Gtex/ } @s +trings; ], }); __END__ Rate one two three one 127000/s -- -21% -22% two 161678/s 27% -- -1% three 163855/s 29% 1% --` [download] This is still quite a bit different to the performance with the initializations included: use strict; use warnings; use Benchmark; our @strings = qw(exception:tex exception:mex asdf tex:exception:mex); Benchmark::cmpthese( -5, { 'one' => q[ my @unfiltered = @strings; my @filtered = grep { /ex +ception:(?!tex)/} @unfiltered; ], 'two' => q[ my @unfiltered = @strings; my @filtered = grep { /ex +ception/ && !/tex/ } @unfiltered; ], 'three' => q[ my @unfiltered = @strings; my @filtered = grep { /ex +ception:/g && !/\Gtex/ } @unfiltered; ], }); __END__ Rate three one two three 81985/s -- -14% -24% one 95390/s 16% -- -12% two 108473/s 32% 14% -- [download] so this illustrates that even simple initialization can have significant impact on results, and I imagine cases where it might be much more (e.g. setting up initial conditions for a database query, where the database itself must be initialized each time, and the connection to it, and caches flushed, etc.).	[reply] [d/l] [select]

Replies are listed 'Best First'.

Re^2: Initializing iterations while benchmarking
by ig (Vicar) on Aug 07, 2009 at 10:34 UTC

Thanks BrowserUk but in your example @unfiltered is empty.

use strict;
use warnings;
use Benchmark;

our @strings = qw(exception:tex exception:mex asdf tex:exception:mex);

Benchmark::cmpthese( -5, {
    'one'   => q[ my @filtered = grep { /exception:(?!tex)/} @strings;
+ ],
    'two'   => q[ my @filtered = grep { /exception/ && !/tex/ } @strin
+gs; ],
    'three' => q[ my @filtered = grep { /exception:/g && !/\Gtex/ } @s
+trings; ],
});
__END__
          Rate   one   two three
one   127000/s    --  -21%  -22%
two   161678/s   27%    --   -1%
three 163855/s   29%    1%    --
[download]

This is still quite a bit different to the performance with the initializations included:

use strict;
use warnings;
use Benchmark;

our @strings = qw(exception:tex exception:mex asdf tex:exception:mex);

Benchmark::cmpthese( -5, {
    'one'   => q[ my @unfiltered = @strings; my @filtered = grep { /ex
+ception:(?!tex)/} @unfiltered; ],
    'two'   => q[ my @unfiltered = @strings; my @filtered = grep { /ex
+ception/ && !/tex/ } @unfiltered; ],
    'three' => q[ my @unfiltered = @strings; my @filtered = grep { /ex
+ception:/g && !/\Gtex/ } @unfiltered; ],
});
__END__
          Rate three   one   two
three  81985/s    --  -14%  -24%
one    95390/s   16%    --  -12%
two   108473/s   32%   14%    --
[download]

so this illustrates that even simple initialization can have significant impact on results, and I imagine cases where it might be much more (e.g. setting up initial conditions for a database query, where the database itself must be initialized each time, and the connection to it, and caches flushed, etc.).

[reply]
[d/l]
[select]