perltutorial
turnstep
<H2>Benchmarking your code</H2>
<H3>What is benchmarking?</H3>
<P>Benchmarking is a way of measuring something - in this
case, how fast your code runs. This is particularly useful
when you are trying to compare two or more ways to do the
same thing, and you want to see which one is faster. You are
really measuring which way is more efficient for perl
to do - the less work it takes perl, the faster it is.</P>
<H3>Why benchmark?</H3>
<P>Small differences add up. A slight change in a small
section of your code may make a big difference, especially
if that code has to perform a lot of work. For example,
a different ways of sorting a collection of words may
not matter much for 100 words, but if you have 100,000
words, the small differences start to matter. Also, it
can be matter of style to make your code as efficient as
possible. It is a good goal to aim for.</P>
<H3>How to benchmark your code</H3>
<P>Benchmarking is usually done with the (surprise) Benchmark
module. This module is very standard, and is very likely
already installed on your system. If not, grab it off
of CPAN.</P>
<P>Benchmarking is not as simple as subtracting the results of
one <STRONG>time</STRONG> call from another one - these are only
accurate to one second, and are not a very good measure. The
benchmark module uses the <STRONG>time</STRONG> function as well
as the <STRONG>times</STRONG> function, which allows for a
much finer measurement of milliseconds.</P>
<P>Here is a quick overview of the Benchmark module:</P>
<P>To use it, just type:
<CODE>
use Benchmark;
</CODE>
at the top of your code. Benchmark has three simple routines
for you to use: <STRONG>timeit</STRONG>,
<STRONG>timethis</STRONG>, and <STRONG>timethese</STRONG>.
Each one needs to know what code to run (sent as the string
<STRONG>CODE</STRONG> in the examples below), as well as how
many times to loop through the code (<STRONG>$count</STRONG>
in the examples below).</P>
<P>For a simple measurement of one piece of code, just use
<STRONG>timeit</STRONG>. You will also need the
<STRONG>timestr</STRONG> routine, which changes the times
that Benchmark uses to a more useful string:
<CODE>
$x = timeit($count, 'CODE');
## CODE is run $count times
## $x becomes a Benchmark object which contains the result
print "Result from $count loops: ";
print timestr($x), "\n";
</CODE></P>
<P>This can be a bit awkward, so Benchmark also has the
<STRONG>timethis</STRONG> routine, which does the same thing
as timeit, but also outputs the results. No timestr is needed
this time:
<CODE>
$x = timethis($count, 'CODE');
## or even just:
timethis($count, 'CODE');
</CODE></P>
<P>The last routine is <STRONG>timethese</STRONG>, which is
the most useful, as it allows you to compare 2 or more
chunks of code at the same time. The syntax is as follows:
<CODE>
@x = timethese($count, { 'one','CODE1', 'two','CODE2' });
</CODE></P>
<P>It returns an array, but this is often unused. Use of the
'alternative comma' is also recommended, to make it easier
to read:
<CODE>
timethese($count, {
'one' => 'CODE1',
'two' => 'CODE2',
'pizza' => 'CODE_X', ## etc....
});
</CODE></P>
<P>It will run each code in the list, and report the result
with the label before it. See the example below for some
sample output.</P>
<P>A final routine to know is <STRONG>timediff</STRONG>
which simply computes the difference between two
Benchmark objects:
<CODE>
$x = timeit($count, 'CODE1');
$y = timeit($count, 'CODE2');
$mydiff = timediff($x, $y);
</CODE></P>
<P>The benchmark module has a few other features, but these
are beyond this tutorial - if interested, check it out
yourself: Benchmark.pm has embedded POD inside it.</P>
<H3>Benchmark Example</H3>
<P>For a simple example of benchmarking, let's compare two
different ways of sorting a list of words. One way will use the
<STRONG>cmp</STRONG> operator, and one will use the
<STRONG><=></STRONG> operator. Which one is faster
for a simple list of words? We will us benchmarking to
find out. For this example, we will create a random list of
1000 words with 6 letters each. Then we'll sort the list
both ways and compare the results.
Here is our complete code:
<CODE>
#!/usr/bin/perl
use Benchmark;
$count = shift || die "Need a count!\n";
## Create a dummy list of 1000 random 6 letter words
srand();
for (1..1000) {
push(@words,
chr(rand(26)+65) . chr(rand(26)+65) . chr(rand(26)+65) .
chr(rand(26)+65) . chr(rand(26)+65) . chr(rand(26)+65));
}
## Method number one - a numeric sort
sub One {
@temp = sort {$a <=> $b} @words;
}
## Method number two - an alphabetic sort
sub Two {
@temp = sort {$a cmp $b} @words;
}
## We'll test each one, with simple labels
timethese (
$count,
{'Method One' => '&One',
'Method Two' => '&Two'}
);
exit;
</CODE></P>
<P>Notice that we store the results of our sort into an
unused variable, @temp, so that @words itself is never
sorted, as we need to use it again.</P>
<P>Here is the result of running it with a count of 10:
<CODE>
Benchmark: timing 10 iterations of Sort One, Sort Two...
Sort One: 0 secs ( 0.33 usr 0.00 sys = 0.33 cpu)
(warning: too few iterations for a reliable count)
Sort Two: 1 secs ( 0.48 usr 0.01 sys = 0.49 cpu)
</CODE></P>
<P>The results tell us four numbers for each code. Notice
that it also gave us a warning for the first one. This
warning is only a guideline, but it is usually right -
we need a higher count. Try to get the number of
cpu seconds (the last number) to be at least 3 seconds or
more for one of the measurements. In our example, let's
try boosting the count to 150:
<CODE>
Benchmark: timing 150 iterations of Sort One, Sort Two...
Sort One: 5 secs ( 4.89 usr 0.01 sys = 4.90 cpu)
Sort Two: 8 secs ( 7.12 usr 0.01 sys = 7.13 cpu)
</CODE></P>
<P>Much better! No warning, and some real times are generated.
Let's look at each of the numbers. The first number is
the elapsed time, or how many seconds the loops took by using
the <STRONG>time</STRONG> function. This is not a very reliable
number: as you can see, with 10 loops, one of the results was
0 seconds. Generally, you can ignore this one, except as a
rough guideline. In particular, a reading of '0' or '1' is
almost useless. Aim for at least an elapsed time of 5 seconds
or more for the best results.</P>
<P>The next three numbers come from the function
<STRONG>times</STRONG>, which returns much more detailed
information. The first two numbers return the user and system
time. Don't be surprised if the system time is often "0" or
very low. These are not as important as the final value, the
cpu time, which is what we are really interested in. This is
the one you should use to make your comparisons. Try to get
at least one of the numbers over 5 seconds - the higher the
number, the more accurate your comparison will be. In this case,
we can see that Method One, the <STRONG><=></STRONG>
operator, is faster at 4.90 cpu seconds compared to the
7.13 seconds that <STRONG>cmp</STRONG> took.</P>
<H3>Tips and Tricks</H3>
<P>Here are some things to think about and watch out for:</P>
<UL>
<LI>Make sure your code works before you start looping it!
This is often overlooked when you are in a hurry. Test it
once with some results and then benchmark it.</LI>
<LI>Add the count to the command line. Something as simple as:
<CODE>
$count = shift || die "Need a count!\n";
</CODE>
keeps you from editing the code every time to try a new
count value.</LI>
<LI>Beware of changes in your repeated loop. Don't change any
variables that are used the next time the loop is run. In
other words, make sure that when you benchmark a chunk of
code, the first loop does exactly the same thing as the last.</LI>
<LI>Move everything out of the loop that you can. You want to
only test what is important. Move things like opening file
handles and initializing values out of the loop. You don't
want to reopen your file 5000 times! Do it once, outside of
the loop.</LI>
<LI>Minimize the test. Similar to the above, try to compare as
few things as possible. A subroutine that slices, sorts,
replaces, and does ten other things will not tell you how
fast each of them is, only how they work together. Change
one thing at a time when comparing two chunks of code.</LI>
<LI>Put the benchmark code at the top of your code. It's
temporary, easy to find, and easy to remove once you are
done testing.</LI>
<LI>Use subroutines to test your code. It keeps the Benchmark
routines uncluttered, and it is easy to make changes to
your subroutines. If the code is really simple, of course,
you can just put the whole code into the argument for
the Benchmark routine</LI>
<LI>Start with a low count, and work your way up. It is
often hard to tell exactly how long the code will take -
so err on the low side. Start with 10, and then move up to
100, then a 1000, then perhaps 5000. You'll get a feel for
it as you go. Aim for at least 5 seconds of elapsed time,
and at least 3 seconds of cpu time. Complicated code and
slow machines may take over a minute to run 100 loops,
while very simple code and very fast machines may require
counts in the millions!</LI>
<LI>Swap the order of your tests around, to make sure that one
is not affecting the other inadvertently. The results
should be the same.</LI>
</UL>