fuzzyping has asked for the wisdom of the Perl Monks concerning the following question:

I've recently been pondering the pros and cons of using perl references. Besides all the neat-o things you can do with various scalar/array/hash/sub refs, it occurred to me that I haven't seen any benchmarking to compare perl references versus their standard variable counterparts.

In order to test, I created a simple sort subroutine inside the Benchmark::timeit call. I've bumped the $count up to 2000 to really get some decent feedback. The testfile is simply a 674-line listing of path/filenames from a tarball.

What I found next wasn't particularly exciting. References (while running under strict) were slightly faster than standard variables (if at all). Indeed, occassionally the test results would come back identical for both samples. However, my next example proved quite interesting...

I decided to run the same script, *without* utilizing the strict pragma (yeah, I know, shame on me). Nevertheless, I just wanted to cover all my bases. The samples now put the non-strict test subject at a 10% faster clip than the reference.

I've included all code and a snapshot of the results below. I'm hoping someone can provide more insight as to my findings (as idiotic as they may be) and suggest further test code. I anticipate that I'm probably going to notice great variations in the results based on the type of code I'm benchmarking. Nevertheless, I would like to find a pattern that I can use to determine at what times it's in my best interest to utilize (or ignore) perl references. TIA!

-fp
#!/usr/bin/perl # benchmark_test_strict.pl use Benchmark; use strict; my @test_array; open(IN, "testfile"); while (<IN>) { chomp; push(@test_array, $_); } close(IN); my $test_ref = \@test_array; my $t0 = timeit(2000, sub { my @var_results = sort {$a cmp $b} @test_a +rray; }); my $t1 = timeit(2000, sub { my @ref_results = sort {$a cmp $b} @$test_ +ref; }); my $td = timediff($t0, $t1); print "\n"; print "normal var (t0)\n"; print "===============\n"; print timestr($t0), "\n\n"; print "reference (t1)\n"; print "==============\n"; print timestr($t1), "\n\n"; print "difference (t0, t1)\n"; print "===================\n"; print timestr($td), "\n\n";
------------------------------------------------------------
[fuzzy@jason fuzzy]$ perl benchmark_test_strict.pl normal var (t0) =============== 3 wallclock secs ( 3.11 usr + 0.00 sys = 3.11 CPU) @ 643.09/s (n=20 +00) reference (t1) ============== 3 wallclock secs ( 3.10 usr + 0.00 sys = 3.10 CPU) @ 645.16/s (n=20 +00) difference (t0, t1) =================== 0 wallclock secs ( 0.01 usr + 0.00 sys = 0.01 CPU)
------------------------------------------------------------
#!/usr/bin/perl # benchmark_test.pl use Benchmark; open(IN, "testfile"); while (<IN>) { chomp; push(@test_array, $_); } close(IN); $test_ref = \@test_array; $t0 = timeit(2000, sub { @var_results = sort {$a cmp $b} @test_array; +}); $t1 = timeit(2000, sub { @ref_results = sort {$a cmp $b} @$test_ref; } +); $td = timediff($t0, $t1); print "\n"; print "normal var (t0)\n"; print "===============\n"; print timestr($t0), "\n\n"; print "reference (t1)\n"; print "==============\n"; print timestr($t1), "\n\n"; print "difference (t0, t1)\n"; print "===================\n"; print timestr($td), "\n\n";
------------------------------------------------------------
[fuzzy@jason fuzzy]$ perl benchmark_test.pl normal var (t0) =============== 4 wallclock secs ( 3.12 usr + 0.00 sys = 3.12 CPU) @ 641.03/s (n=20 +00) reference (t1) ============== 3 wallclock secs ( 3.50 usr + 0.01 sys = 3.51 CPU) @ 569.80/s (n=20 +00) difference (t0, t1) =================== 1 wallclock secs (-0.38 usr + -0.01 sys = -0.39 CPU)

Replies are listed 'Best First'.
Re: Performance of Perl references
by perrin (Chancellor) on Jul 22, 2002 at 07:13 UTC
    I think you're not understanding what references do. In your example, you are not comparing slinging a bunch of data around vs. using a reference, you are comparing directly accessing a lexical variable vs. de-referencing the same variable and then accessing. I would expect the de-ref to always be slower.

    There are tons of reasons to use references, but if you want to see the speed advantages of them, bench something like this:

    #!/usr/bin/perl -w use Benchmark; my @array; for (0..1000) { $array[$_] = rand(1000); } timethese(200, { 'ref' => sub { my @unsorted = @array; my $return_ref = sort_array_ref(\@unsorted); }, 'old skool' => sub { my @unsorted = @array; my @return_array = sort_array(@unsorted); }, }); sub sort_array_ref { my $array_ref = shift; my @sorted_array = sort @{$array_ref}; return \@sorted_array; } sub sort_array { my @array = @_; my @sorted_array = sort @array; return @sorted_array; } __END__ Benchmark: timing 200 iterations of old skool, ref... old skool: 6 wallclock secs ( 5.81 usr + 0.01 sys = 5.82 CPU) @ 34 +.36/s (n=200) ref: 6 wallclock secs ( 5.07 usr + 0.01 sys = 5.08 CPU) @ 39 +.37/s (n=200)
Re: Performance of Perl references
by kjherron (Pilgrim) on Jul 22, 2002 at 07:27 UTC
    Like Perrin says, your benchmark functions aren't well suited to what you're testing. Your functions only look up the array to be sorted once per invocation; then you go on to sort the array and then assign the result into another variable. The cost of using a reference is going to be swamped by these later steps.

    The big performance win from references comes from being able to pass them in and out of subroutines more efficiently. This benchmark illustrates the benefit more clearly:

    use strict; use Benchmark; my @x = (0..100_000); sub pass_by_value(@) { my $x; foreach (@_) { $x += $_ } return $x + } sub pass_by_reference(\@) { my $x; foreach (@$_) { $x += $_ } return $ +x } timethese(500, { 'Pass by Ref' => sub { pass_by_reference(@x) }, 'Pass by Value' => sub { pass_by_value(@x) } }); sub return_by_value(@) { return @x }; sub return_by_reference(\@) { return \@x }; timethese(500, { 'Return by Ref' => sub { return_by_reference(@x) }, 'Return by Value' => sub { return_by_value(@x) } });
    Here are the benchmark results on my system:
    Benchmark: timing 500 iterations of Pass by Ref, Pass by Value...
    Pass by Ref:  0 wallclock secs ( 0.00 usr +  0.00 sys =  0.00 CPU)
                (warning: too few iterations for a reliable count)
    Pass by Value: 78 wallclock secs (67.48 usr +  0.17 sys = 67.65 CPU) @  7.39/s (n=500)
    Benchmark: timing 500 iterations of Return by Ref, Return by Value...
    Return by Ref:  0 wallclock secs ( 0.00 usr +  0.00 sys =  0.00 CPU)
                (warning: too few iterations for a reliable count)
    Return by Value: 16 wallclock secs (14.19 usr +  0.03 sys = 14.22 CPU) @ 35.16/s (n=500)
    
      So, passing references is where I'm going to experience the most performance gains from using refs. Makes sense. Hindsight, my benchmark was highly flawed in what it attempted to quantify. However, the real crux of my post has been, for the most part, left unanswered... where (else) are you likely to note great gains from using references. Where do you not want to use references?

      -fp
        Any time you derefernce to get to a value pointed to by a reference, it's called a layer of indirection. Every layer of indirection adds some overhead, so if you are acccessing data deep inside a complex data structure, it adds up. Normally this is negligable, but if you are doing so inside loops you can gain more speed by removing as many layers of indirecion as possible.
        Take the following code sample. The structure '$data' is two levels deep, and the subs 'foo' and 'bar' both loop thru and access all the data at the deepest level. 'foo' uses a naive aproach of multiple layers of indirection at the deepest level 'my $val = $data->{$key1}->{$key2}', while 'bar' reduces the indirection by assigning the inner data struct to a variable 'my $data2 = $data->{$key1}' and looping over it, accessing that data with 'my $val = $data2->{$key2}'.

        results:

        Benchmark: timing 100000 iterations of multiple_indirection, reduced_indirection...
        multiple_indirection: 29 wallclock secs (28.08 usr +  0.00 sys = 28.08 CPU) @ 3561.25/s (n=100000)
        reduced_indirection: 25 wallclock secs (24.19 usr +  0.00 sys = 24.19 CPU) @ 4133.94/s (n=100000)
                               Rate multiple_indirection  reduced_indirection
        multiple_indirection 3561/s                   --                 -14%
        reduced_indirection  4134/s                  16%                   --
        

        code:
        #!/usr/bin/perl my %innerdata; @innerdata{a..z} = a..z; my $data = { foo => { %innerdata }, bar => { %innerdata } }; sub foo { my $data = $_[0]; for $key1 (keys %$data) { for $key2 (keys %{$data->{$key1}}) { my $val = $data->{$key1}->{$key2}; # process $val; # print $val; } } } sub bar { my $data = $_[0]; for $key1 (keys %$data) { my $data2 = $data->{$key1}; for $key2 (keys %$data2) { my $val = $data2->{$key2}; # process $val; # print $val; } } } use Benchmark qw(cmpthese); cmpthese( 100000, { multiple_indirection => sub { foo($data) }, reduced_indirection => sub { bar($data) }, } )
        -- O thievish Night, Why should'st thou, but for some felonious end, In thy dark lantern thus close up the stars? --Milton
Re: Performance of Perl references
by grinder (Bishop) on Jul 22, 2002 at 08:41 UTC
    In order to test, I created a simple sort subroutine inside the Benchmark::timeit call. I've bumped the $count up to 2000 to really get some decent feedback.

    I would tend to multiply the value of $count by 50 to a 100 or more. 3 seconds is barely enough time to compare anything. 0.39 seconds difference in CPU time is mere noise.

    Remember that a difference of 5 to 10 percent in benchmark performance can also be considered noise. Hence you would want to see 30 seconds vs. 35 seconds (for example) in order to conclude a real (although minor) difference. 30 seconds vs. 75 seconds... now you're starting to talk about something interesting.


    print@_{sort keys %_},$/if%_=split//,'= & *a?b:e\f/h^h!j+n,o@o;r$s-t%t#u'
Re: Performance of Perl references
by BUU (Prior) on Jul 22, 2002 at 07:42 UTC
    Also note that 'use strict;' itself carries a fairly heavy performance penalty, which would easily account for the 10% difference your seeing.