No More Meaningless Benchmarks!

In Use of single quotes versus double quotes, the original poster asks "Which is faster, single quotes or double quotes?"

As you might expect, a couple of people responded with "It doesn't matter!" grinder followed up with more detail, explaining exactly why it doesn't matter to Perl. In this case, Perl compiles a non-interpolating double-quoted string and a single-quoted string (which never interpolates) into the exact same opcodes.

That is, Perl does the same work.

As you might expect, another monk tried to be helpful by pulling out Benchmark. Re: Use of single quotes versus double quotes argues that, on one machine, single-quoted constant non-interpolating strings are 16% slower than double-quoted strings.

Yet that's completely meaningless. Why? Because it has nothing to do with Perl! As grinder demonstrated, perl performs the same actions for each type of string. The differences are elsewhere in the system. It could be that one subroutine has ops all within one memory page and the other has less locality. Looking at the length of the strings, I suspect that there's a buffer size issue as well (IO screws up all benchmarks). Who knows if there are other processes running?

For fun, I started with a modified benchmark that runs for only five seconds and uses warn to send all output to STDERR, so I could redirect it to /dev/null. Here's my start:

#!/usr/bin/perl

use strict;
use warnings;

use Benchmark qw(cmpthese);
my $tests = -5;
sub single { warn 'This is a test'; }
sub double { warn "This is a test"; }
cmpthese(
    $tests, 
    { 
        single => \&single,
        double => \&double
    }
);
[download]

... and the output:

$ perl bm_string_quoting.pl 2> /dev/null
           Rate single double
single 476308/s     --    -1%
double 481758/s     1%     --
[download]

What happened if I changed the order of subroutine declarations? That might change the memory layout:

#!/usr/bin/perl

use strict;
use warnings;

use Benchmark qw(cmpthese);
my $tests = -5;
sub double { warn "This is a test"; }
sub single { warn 'This is a test'; }
cmpthese(
    $tests, 
    { 
        single => \&single,
        double => \&double
    }
);
[download]

The output is:

$ perl bm_string_quoting.pl 2> /dev/null
           Rate double single
double 474650/s     --    -2%
single 483537/s     2%     --
[download]

Would removing strict and warnings have a similar effect?

#!/usr/bin/perl

use Benchmark qw(cmpthese);
my $tests = -5;
sub double { warn "This is a test"; }
sub single { warn 'This is a test'; }
cmpthese(
    $tests, 
    { 
        single => \&single,
        double => \&double
    }
);
[download]

Indeed:

perl bm_string_quoting.pl 2> /dev/null
           Rate single double
single 491116/s     --    -2%
double 502236/s     2%     --
[download]

How about moving the declarations further?

#!/usr/bin/perl

use Benchmark qw(cmpthese);
my $tests = -5;
sub single { warn 'This is a test'; }
cmpthese(
    $tests, 
    { 
        single => \&single,
        double => \&double
    }
);
sub double { warn "This is a test"; }
[download]

Interesting:

$ perl bm_string_quoting.pl 2> /dev/null
           Rate double single
double 449265/s     --    -7%
single 481758/s     7%     --
[download]

There's one more code movement to try:

#!/usr/bin/perl

use Benchmark qw(cmpthese);
my $tests = -5;
cmpthese(
    $tests, 
    { 
        single => \&single,
        double => \&double
    }
);
sub double { warn "This is a test"; }
sub single { warn 'This is a test'; }
[download]

As expected, the results are unpredictable:

$ perl bm_string_quoting.pl 2> /dev/null
           Rate single double
single 459485/s     --    -0%
double 460340/s     0%     --
[download]

There are two possible morals to this story. The first is that moving code around can make your programs much faster. A better moral is know when benchmarks are useless.

Back to Meditations