in reply to Re: Benchmark.pm: Does subroutine testing order bias results?
in thread Benchmark.pm: Does subroutine testing order bias results?

For what it's worth... 

Benchmark: timing 5 iterations of Atest, Btest, Ctest, Dtest...
     Atest:  4 wallclock secs ( 4.08 usr +  0.23 sys =  4.31 CPU) @  1
+.16/s (n=5)
     Btest:  5 wallclock secs ( 4.18 usr +  0.00 sys =  4.18 CPU) @  1
+.20/s (n=5)
     Ctest:  5 wallclock secs ( 4.17 usr +  0.00 sys =  4.17 CPU) @  1
+.20/s (n=5)
     Dtest:  5 wallclock secs ( 4.21 usr +  0.00 sys =  4.21 CPU) @  1
+.19/s (n=5)
        Rate Atest Dtest Btest Ctest
Atest 1.16/s    --   -2%   -3%   -3%
Dtest 1.19/s    2%    --   -1%   -1%
Btest 1.20/s    3%    1%    --   -0%
Ctest 1.20/s    3%    1%    0%    --

[download]
Update: possibly more important, the problem isn't exaserbated by increases iterations... 
laptop:~> monk.pl
Benchmark: timing 20 iterations of Atest, Btest, Ctest, Dtest...
     Atest: 17 wallclock secs (16.61 usr +  0.25 sys = 16.86 CPU) @  1
+.19/s (n=20)
     Btest: 16 wallclock secs (16.73 usr +  0.00 sys = 16.73 CPU) @  1
+.20/s (n=20)
     Ctest: 17 wallclock secs (16.72 usr +  0.00 sys = 16.72 CPU) @  1
+.20/s (n=20)
     Dtest: 17 wallclock secs (16.71 usr +  0.02 sys = 16.73 CPU) @  1
+.20/s (n=20)
        Rate Atest Btest Dtest Ctest
Atest 1.19/s    --   -1%   -1%   -1%
Btest 1.20/s    1%    --   -0%   -0%
Dtest 1.20/s    1%    0%    --   -0%
Ctest 1.20/s    1%    0%    0%    --

[download]
System Info... 
laptop:~> uname -a
Linux laptop 2.4.25-1-686 #1 Thu Apr 15 15:21:42 CEST 2004 i686 unknow
+n
laptop:~> perl -V
Summary of my perl5 (revision 5.0 version 6 subversion 1) configuratio
+n:
  Platform:
    osname=linux, osvers=2.6.3-deb2-skas3, archname=i386-linux
    uname='linux mizar 2.6.3-deb2-skas3 #1 sun mar 14 14:46:35 pst 200
+4 i686 unknown '
    config_args='-Dccflags=-DDEBIAN -Dcccdlflags=-fPIC -Darchname=i386
+-linux -Dprefix=/usr -Dprivlib=/usr/share/perl/5.6.1 -Darchlib=/usr/l
+ib/perl/5.6.1 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvend
+orarch=/usr/lib/perl5 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/sh
+are/perl/5.6.1 -Dsitearch=/usr/local/lib/perl/5.6.1 -Dman1dir=/usr/sh
+are/man/man1 -Dman3dir=/usr/share/man/man3 -Dman1ext=1 -Dman3ext=3per
+l -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Uusesfio -Duseshrpli
+b -Dlibperl=libperl.so.5.6.1 -Dd_dosuid -des'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=undef use5005threads=undef useithreads=undef usemultipl
+icity=undef
    useperlio=undef d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=undef use64bitall=undef uselongdouble=undef
  Compiler:
    cc='cc', ccflags ='-DDEBIAN -fno-strict-aliasing -I/usr/local/incl
+ude -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2',
    cppflags='-DDEBIAN -fno-strict-aliasing -I/usr/local/include'
    ccversion='', gccversion='2.95.4 20011002 (Debian prerelease)', gc
+cosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=1
+2
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
+ lseeksize=8
    alignbytes=4, usemymalloc=n, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-lgdbm -ldb -ldl -lm -lc -lcrypt
    perllibs=-ldl -lm -lc -lcrypt
    libc=/lib/libc-2.2.5.so, so=so, useshrplib=true, libperl=libperl.s
+o.5.6.1
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-rdynami
+c'
    cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib'


Characteristics of this binary (from libperl): 
  Compile-time options: USE_LARGE_FILES
  Built under linux
  Compiled at Apr  4 2004 05:57:53
  @INC:
    /usr/local/lib/perl/5.6.1
    /usr/local/share/perl/5.6.1
    /usr/lib/perl5
    /usr/share/perl5
    /usr/lib/perl/5.6.1
    /usr/share/perl/5.6.1
    /usr/local/lib/site_perl
    .

[download]

Replies are listed 'Best First'.
Re^3: Benchmark.pm: Does subroutine testing order bias results? (Thanks!)
by BrowserUk (Patriarch) on Jul 15, 2004 at 05:17 UTC
    For what it's worth...

    Actually, it worth quite a lot--to me at least. Thankyou greatly.

    Your post prompted me to try my benchmark with 5.6.1 and the difference is stark. 

    C:\>perl5.6.1 p:\test\373536-2.pl
[SNIP]
        Rate Btest Atest Ctest Dtest
Btest 9.14/s    --   -0%   -3%   -3%
Atest 9.14/s    0%    --   -3%   -3%
Ctest 9.40/s    3%    3%    --   -0%
Dtest 9.42/s    3%    3%    0%    --

C:\>perl5.8.4 p:\test\373536-2.pl
      s/iter Btest Dtest Ctest Atest
Btest   4.23    --   -0%   -0%  -18%
Dtest   4.22    0%    --   -0%  -18%
Ctest   4.22    0%    0%    --  -18%
Atest   3.45   22%   22%   22%    --

    [download]

    Not only is does the first-run bias disappear, but look at the iteration times! With 5.6.1, they are in iterations per second. For 5.8.4 thay are in seconds per iteration. If my math is up to it this morning, I make that 5.8.4 is 40x slower than 5.6.1!...

    That's not the end of the story. The 5.6.1 involved is home built from the CPAN distribution. The 5.8.4 involved is AS 810.

    I've royally screwed my home-build copy of 5.8.4 trying to get USE_PERL_MALLOC to work, so I haven't been able to test that yet, but I am going to re-unzip the lot and start from scratch.

    All of which is probably of not muct interest to you, but it has made me look again at various things. Again, thankyou.

    Examine what is said, not who speaks.
    "Efficiency is intelligent laziness." -David Dunham
    "Think for yourself!" - Abigail
    "Memory, processor, disk in that order on the hardware side. Algorithm, algoritm, algorithm on the code side." - tachyon
[reply]
[d/l]

In Section Seekers of Perl Wisdom