I spent a good deal of time using Benchmark.pm over the last week and have learned a lot. I hope this provides a diversion that most will find interesting and demonstrates some unusual results of the benchmarking process. I suppose that if I had to single out one of the most abused notions of benchmarking, it would have to be that just because a snippet of code performs the most executions per second in a timethis or cmpthese result doesn't necessarily mean that it will perform the best in real world conditions.

It is quite likely that my results will not match those obtained on different computers, versions of Perl, operating systems or other application mixes, but I can say that all results were run multiple times and then averaged.

1. Which of the following snippets averages the most executions per second?

a) 's/\s*#.*//' b) '$_ = $` if /#/' c) '($_) = split/#/' d) '$_ = substr($_, 0, $-[0]) if /#/' e) 'if ((my $p = index($_, "#")) > -1) { substr($_, $p, -1, "") }'

Answer: d

2. Which snippet from the previous question averages the fewest executions per second?

Answer: c

3. Which of the following snippets can cause a runtime error?

a) '{}' b) '' c) ';' d) All of the above e) None of the above

Answer: If you answered a or c, more certain would be b. If you answered b, correct for most computers. If you answered d, you have a much better computer than mine. If you answered e, your computer might be a collector's item.

4. What runtime error is most likely produced by the previous question?

Answer: Range iterator outside integer range

5. Which of the following snippets averages the most executions per second?

a) '$_ = (split/#/)[0]' b) '($_) = split/#/' c) Too close to tell

Answer: b, by about 9%

6. Which of the following snippets averages the most executions per second?

a) '$i = ""; $i = 1 if ($_ % 2) == 0' b) 'if (($_ % 2) == 0) {$i = 1} else {$i = ""}' c) Too close to tell

Answer: b, by about 4%

7. Which of the following snippets averages the most executions per second?

a) '($str1, $str2) = split/:/ if /:/', b) 'if (/:/) {$str1 = $`; $str2 = $\'}', c) 'if (/:/) {$str1 = substr($_,0, $-[0]); $str2 = substr($_,$+[0]) +}' d) '($str1,$str2) = m/(\d*):(\d*)/ if /:/' e) Too close to call.

Answer: e, allowing for at least 3% error.

8. Which of the snippets in the previous question runs the fastest in the following code?

$_ = "123:45678"; foreach (1..2000000) { # snippet goes here ; }

Answer: e, allowing for at least 3% error.

9. Which of the following snippets averages the most executions per second?
a) 'foreach (1..$n){$m = $_}' b) '$m = $_ for (1..$n)' c) 'for ($i=1;$i<$n+1;$i++) {$m = $i}' d) Too close to call.

Answer: c, by about 10%

10. Which of the snippets from the previous question runs the fastest in the following code?
my ($i, $m); my $n = 5000000; # snippet goes here

Answer: b, by about 7%

--Jim


In reply to Benchmarking Quiz by jlongino

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.