Re^3: Benchmark.pm: Does subroutine testing order bias results? (Thanks!)

For what it's worth...

Actually, it worth quite a lot--to me at least. Thankyou greatly.

Your post prompted me to try my benchmark with 5.6.1 and the difference is stark.

C:\>perl5.6.1 p:\test\373536-2.pl
[SNIP]
        Rate Btest Atest Ctest Dtest
Btest 9.14/s    --   -0%   -3%   -3%
Atest 9.14/s    0%    --   -3%   -3%
Ctest 9.40/s    3%    3%    --   -0%
Dtest 9.42/s    3%    3%    0%    --

C:\>perl5.8.4 p:\test\373536-2.pl
      s/iter Btest Dtest Ctest Atest
Btest   4.23    --   -0%   -0%  -18%
Dtest   4.22    0%    --   -0%  -18%
Ctest   4.22    0%    0%    --  -18%
Atest   3.45   22%   22%   22%    --
[download]

Not only is does the first-run bias disappear, but look at the iteration times! With 5.6.1, they are in iterations per second. For 5.8.4 thay are in seconds per iteration. If my math is up to it this morning, I make that 5.8.4 is 40x slower than 5.6.1!...

That's not the end of the story. The 5.6.1 involved is home built from the CPAN distribution. The 5.8.4 involved is AS 810.

I've royally screwed my home-build copy of 5.8.4 trying to get USE_PERL_MALLOC to work, so I haven't been able to test that yet, but I am going to re-unzip the lot and start from scratch.

All of which is probably of not muct interest to you, but it has made me look again at various things. Again, thankyou.

Examine what is said, not who speaks.

"Efficiency is intelligent laziness." -David Dunham
"Think for yourself!" - Abigail
"Memory, processor, disk in that order on the hardware side. Algorithm, algoritm, algorithm on the code side." - tachyon

Comment on Re^3: Benchmark.pm: Does subroutine testing order bias results? (Thanks!) Download Code