in reply to Unexpected OO accessor benchmarks

Just for fun, I rearranged the order of declaration, putting get_value() and get_value2() ahead of a(), b(), and c() (as in No More Meaningless Benchmarks!). Here are the results of the first run:

$ perl oo_benchmark.pl (warning: too few iterations for a reliable count) (warning: too few iterations for a reliable count) (warning: too few iterations for a reliable count) Rate normal optimized direct normal 142857143/s -- -14% -57% optimized 166666667/s 17% -- -50% direct 333333333/s 133% 100% --

Here are the results of my modified program:

$ perl oo_benchmark.pl (warning: too few iterations for a reliable count) (warning: too few iterations for a reliable count) (warning: too few iterations for a reliable count) Rate direct normal optimized direct 125000000/s -- -25% -25% normal 166666667/s 33% -- 0% optimized 166666667/s 33% 0% --

I conclude that these kinds of benchmarks are meaningless.

Replies are listed 'Best First'.
Re^2: Unexpected OO accessor benchmarks
by eric256 (Parson) on Feb 10, 2007 at 02:24 UTC

    Actually they arn't meaningless. The fact that I have to do 1_000_000 iterations before I can even tell the difference between the different access method shows me that this isn't a good place to try and improve performance. Unless of course you are doing millions and millions and millions of these accesses every second, then you might care, but i'm betting if thats the case you'd better have some beefy hardware anyways ;)


    ___________
    Eric Hodges
Re^2: Unexpected OO accessor benchmarks
by cLive ;-) (Prior) on Feb 10, 2007 at 01:08 UTC

    I read your node - interesting, thanks. But I can't get results like yours. Could it be because the number of iterations in your test are too low?

    Putting get_value() and get_value2() declarations at the beginning of program after the "my $self" line, I get:

    Rate normal optimized direct normal 174625/s -- -27% -60% optimized 240602/s 38% -- -45% direct 435374/s 149% 81% --

    Moving the subs to just above sub a() declaration, I get

    Rate normal optimized direct normal 172275/s -- -28% -61% optimized 239252/s 39% -- -46% direct 444444/s 158% 86% --

    I've tried a few more re-arrangements of the code layout, and I can't get the reversal that you did. All re-arrangements I've tried have 'direct' at 145-155%. Any chance you can post the code so I can duplicate the results? Logically, I would think that asking for the value of an arrayref element is going to be faster than calling a sub to do so, so the benchmarks appear to be logical to me.

      I revised the code slightly:

      use Benchmark 'cmpthese'; sub get_value { my $self = shift; return $self->[0]; } sub get_value2 { $_[0]->[0]; } my $self = bless [123], '::main'; cmpthese( 16000000, { normal => sub { $self->get_value() }, optimized => sub { $self->get_value2() }, direct => sub { $self->[0] }, }); exit(0); $ perl oo_benchmark.pl Rate normal optimized direct normal 1462523/s -- -21% -90% optimized 1858304/s 27% -- -88% direct 15238095/s 942% 720% --

      Yes, it took 16 million iterations to get an accurate reading on the direct attribute access. With a variant of your original benchmark modified to avoid the double-dispatch penalty, I get:

      use Benchmark 'cmpthese'; my $self = bless [123], '::main'; cmpthese( 16000000, { normal => sub { $self->get_value() }, optimized => sub { $self->get_value2() }, direct => sub { $self->[0] }, }); exit(0); sub get_value { my $self = shift; return $self->[0]; } sub get_value2 { $_[0]->[0]; } $ perl oo_benchmark.pl Rate normal optimized direct normal 1725998/s -- -14% -94% optimized 2007528/s 16% -- -93% direct 29629630/s 1617% 1376% --

      That represents speedups of around 20%, 11%, and 100%, respectively. I can make the case for a 10% speedup perhaps, but the others are way outside anything I can explain. Thus I'm not sure this benchmark is useful.