in reply to Re: Benchmark.pm: Does subroutine testing order bias results?
in thread Benchmark.pm: Does subroutine testing order bias results?
1. Across Perls and OSes, increasing the number of iterations in cmpthese() smoothed out the differences in results from the 4 tests in each call. Increasing the number of elements mapped to @strings by test() -- on the 2 OSes where I was able to test that -- didn't have a consistent effect in this regard.
2. On 3 of the 4 Perl/OS combinations on which the test was run, the first run of the test within a given iteration ran more slowly -- albeit slightly -- than subsequent runs. Only on the Perl 5.8.0/Win2K combination was the first run faster than subsequent ones. This is the opposite of what I reported in my OP to this thread, where I found that, while comparing two different versions of a module, the first subroutine tested via timethese() ran faster than the second.
It may be that this outcome is due to the type of work which the subroutine being benchmarked is called upon to do. In BrowserUk's test(), only 1 array is created with each iteration. In my OP, I was creating the heaviest version of a List::Compare object, which internally allocates more than a dozen arrays and hashes. But since this object goes out of scope and is presumably destroyed with each call to timethese(), it's not clear why this necessarily means that the second subroutine should run more slowly than the first.
The plot thickens.
System Infosub test { my @strings = map{ ' ' x 1000 } 1 .. $records; } cmpthese( $iterations, { Atest => \&test, Btest => \&test, Ctest => \&test, Dtest => \&test, } ); __END__ # 1. BrowserUk's &test on Perl 5.8.0 on Win2K # Done 6 ways: # a. Testing 5 iterations of 25000 elements ... Rate Dtest Btest Ctest Atest Dtest 1.04/s -- -0% -0% -13% Btest 1.04/s 0% -- -0% -13% Ctest 1.05/s 0% 0% -- -13% Atest 1.20/s 15% 15% 15% -- # b. Testing 5 iterations of 50000 elements ... s/iter Dtest Btest Ctest Atest Dtest 3.66 -- -0% -0% -20% Btest 3.66 0% -- -0% -20% Ctest 3.66 0% 0% -- -20% Atest 2.93 25% 25% 25% -- # c. Testing 50 iterations of 25000 elements ... Rate Ctest Dtest Btest Atest Ctest 1.03/s -- -0% -0% -2% Dtest 1.03/s 0% -- -0% -2% Btest 1.04/s 0% 0% -- -2% Atest 1.06/s 2% 2% 2% -- # d. Testing 50 iterations of 50000 elements ... s/iter Dtest Ctest Btest Atest Dtest 3.19 -- -0% -0% -2% Ctest 3.18 0% -- -0% -2% Btest 3.18 0% 0% -- -2% Atest 3.12 2% 2% 2% -- # e. Testing 100 iterations of 25000 elements ... Rate Btest Atest Ctest Dtest Btest 1.03/s -- -0% -1% -1% Atest 1.04/s 0% -- -0% -1% Ctest 1.04/s 1% 0% -- -0% Dtest 1.04/s 1% 1% 0% -- # f. Testing 100 iterations of 50000 elements ... s/iter Dtest Ctest Btest Atest Dtest 5.71 -- -0% -0% -1% Ctest 5.71 0% -- -0% -1% Btest 5.69 0% 0% -- -1% Atest 5.65 1% 1% 1% -- # 2. BrowserUk's &test on Perl 5.8.4 on Darwin # Done 6 ways: # a. Testing 5 iterations of 25000 elements ... Rate Atest Dtest Btest Ctest Atest 2.49/s -- -4% -5% -6% Dtest 2.59/s 4% -- -1% -2% Btest 2.62/s 5% 1% -- -1% Ctest 2.65/s 6% 2% 1% -- # b. Testing 5 iterations of 50000 elements ... Rate Atest Btest Ctest Dtest Atest 1.22/s -- -6% -6% -7% Btest 1.29/s 6% -- -0% -1% Ctest 1.29/s 6% -0% -- -1% Dtest 1.30/s 7% 1% 1% -- # c. Testing 50 iterations of 25000 elements ... Rate Atest Dtest Btest Ctest Atest 2.59/s -- -1% -1% -1% Dtest 2.60/s 1% -- -0% -0% Btest 2.60/s 1% 0% -- -0% Ctest 2.61/s 1% 0% 0% -- # d. Testing 50 iterations of 50000 elements ... Rate Atest Btest Ctest Dtest Atest 1.28/s -- -0% -1% -1% Btest 1.29/s 0% -- -0% -0% Ctest 1.29/s 1% 0% -- -0% Dtest 1.29/s 1% 0% 0% -- # e. Testing 100 iterations of 25000 elements ... Rate Ctest Dtest Btest Atest Ctest 2.60/s -- -0% -0% -2% Dtest 2.60/s 0% -- -0% -2% Btest 2.61/s 0% 0% -- -2% Atest 2.65/s 2% 2% 2% -- # f. Testing 100 iterations of 50000 elements ... Rate Atest Btest Dtest Ctest Atest 1.29/s -- -0% -0% -0% Btest 1.29/s 0% -- -0% -0% Dtest 1.29/s 0% 0% -- -0% Ctest 1.29/s 0% 0% 0% -- # 3. BrowserUk's &test on Perl 5.6.0 on RedHat Linux 7.2 # Done 3 ways; was unable to do 50000 elements test due to # excessive swapping to disk # a. Testing 5 iterations of 25000 elements ... Rate Atest Btest Ctest Dtest Atest 1.80/s -- -4% -4% -4% Btest 1.87/s 4% -- -0% -0% Ctest 1.87/s 4% 0% -- 0% Dtest 1.87/s 4% 0% 0% -- # b. Testing 50 iterations of 25000 elements ... Rate Atest Btest Ctest Dtest Atest 1.86/s -- -0% -1% -1% Btest 1.87/s 0% -- -0% -0% Ctest 1.87/s 1% 0% -- 0% Dtest 1.87/s 1% 0% 0% -- # c. Testing 100 iterations of 25000 elements ... Rate Atest Dtest Ctest Btest Atest 1.86/s -- -0% -0% -0% Dtest 1.87/s 0% -- -0% -0% Ctest 1.87/s 0% 0% -- -0% Btest 1.87/s 0% 0% 0% -- # 4. BrowserUk's &test on Perl 5.6.1 on Windows98SE # Done 3 ways; was unable to do 50000 elements test due to # excessive swapping to disk # a. Testing 5 iterations of 25000 elements ... Rate Atest Dtest Btest Ctest Atest 2.02/s -- -9% -9% -11% Dtest 2.22/s 10% -- -0% -2% Btest 2.22/s 10% 0% -- -2% Ctest 2.27/s 12% 2% 2% -- # c. Testing 50 iterations of 25000 elements ... Rate Atest Dtest Ctest Btest Atest 2.20/s -- -1% -1% -1% Dtest 2.22/s 1% -- -0% -0% Ctest 2.22/s 1% 0% -- 0% Btest 2.22/s 1% 0% 0% -- # e. Testing 100 iterations of 25000 elements ... Rate Atest Dtest Ctest Btest Atest 2.19/s -- -1% -1% -2% Dtest 2.21/s 1% -- -0% -0% Ctest 2.21/s 1% 0% -- -0% Btest 2.22/s 2% 0% 0% --
# Win2K Summary of my perl5 (revision 5 version 8 subversion 0) configuration: Platform: osname=MSWin32, osvers=4.0, archname=MSWin32-x86-multi-thread uname='' config_args='undef' hint=recommended, useposix=true, d_sigaction=undef usethreads=undef use5005threads=undef useithreads=define usemultip +licity=def ine useperlio=define d_sfio=undef uselargefiles=define usesocks=undef use64bitint=undef use64bitall=undef uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cl', ccflags ='-nologo -Gf -W3 -MD -Zi -DNDEBUG -O1 -DWIN32 -D +_CONSOLE - DNO_STRICT -DHAVE_DES_FCRYPT -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_ +SYS -DUSE_ PERLIO -DPERL_MSVCRT_READFIX', optimize='-MD -Zi -DNDEBUG -O1', cppflags='-DWIN32' ccversion='', gccversion='', gccosandvers='' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234 d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=10 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='__int64 +', lseeksi ze=8 alignbytes=8, prototype=define Linker and Libraries: ld='link', ldflags ='-nologo -nodefaultlib -debug -opt:ref,icf -l +ibpath:"L: \Perl\lib\CORE" -machine:x86' libpth="L:\Perl\lib\CORE" libs= oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib + comdlg32 .lib advapi32.lib shell32.lib ole32.lib oleaut32.lib netapi32.lib uui +d.lib wsoc k32.lib mpr.lib winmm.lib version.lib odbc32.lib odbccp32.lib msvcrt. +lib perllibs= oldnames.lib kernel32.lib user32.lib gdi32.lib winspool +.lib comd lg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib netapi32.lib + uuid.lib wsock32.lib mpr.lib winmm.lib version.lib odbc32.lib odbccp32.lib msv +crt.lib libc=msvcrt.lib, so=dll, useshrplib=yes, libperl=perl58.lib gnulibc_version='undef' Dynamic Linking: dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' ' cccdlflags=' ', lddlflags='-dll -nologo -nodefaultlib -debug -opt: +ref,icf - libpath:"L:\Perl\lib\CORE" -machine:x86' Characteristics of this binary (from libperl): Compile-time options: MULTIPLICITY USE_ITHREADS USE_LARGE_FILES PERL +_IMPLICIT_ CONTEXT PERL_IMPLICIT_SYS Locally applied patches: ActivePerl Build 806 Built under MSWin32 Compiled at Mar 31 2003 00:45:44 @INC: L:/Perl/lib L:/Perl/site/lib . #Darwin Summary of my perl5 (revision 5 version 8 subversion 4) configuration: Platform: osname=darwin, osvers=7.4.0, archname=darwin-2level uname='darwin james-e-keenans-computer.local 7.4.0 darwin kernel v +ersion 7.4.0: wed may 12 16:58:24 pdt 2004; root:xnuxnu-517.7.7.obj~7 +release_ppc power macintosh powerpc ' config_args='-de' hint=recommended, useposix=true, d_sigaction=define usethreads=undef use5005threads=undef useithreads=undef usemultipl +icity=undef useperlio=define d_sfio=undef uselargefiles=define usesocks=undef use64bitint=undef use64bitall=undef uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cc', ccflags ='-pipe -fno-common -DPERL_DARWIN -no-cpp-precomp + -fno-strict-aliasing', optimize='-Os', cppflags='-no-cpp-precomp -pipe -fno-common -DPERL_DARWIN -no-cpp- +precomp -fno-strict-aliasing' ccversion='', gccversion='3.3 20030304 (Apple Computer, Inc. build + 1495)', gccosandvers='' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=4321 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=8 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', + lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='env MACOSX_DEPLOYMENT_TARGET=10.3 cc', ldflags ='' libpth=/usr/lib libs=-ldbm -ldl -lm -lc perllibs=-ldl -lm -lc libc=/usr/lib/libc.dylib, so=dylib, useshrplib=false, libperl=libp +erl.a gnulibc_version='' Dynamic Linking: dlsrc=dl_dyld.xs, dlext=bundle, d_dlsymun=undef, ccdlflags=' ' cccdlflags=' ', lddlflags=' -bundle -undefined dynamic_lookup' Characteristics of this binary (from libperl): Compile-time options: USE_LARGE_FILES Built under darwin Compiled at Jun 2 2004 09:36:19 @INC: /usr/local/lib/perl5/5.8.4/darwin-2level /usr/local/lib/perl5/5.8.4 /usr/local/lib/perl5/site_perl/5.8.4/darwin-2level /usr/local/lib/perl5/site_perl/5.8.4 /usr/local/lib/perl5/site_perl . #Linux Summary of my perl5 (revision 5.0 version 6 subversion 0) configuratio +n: Platform: osname=linux, osvers=2.4.6-3.1enterprise, archname=i386-linux uname='linux stripples.devel.redhat.com 2.4.6-3.1enterprise #1 smp + tue jul 24 14:03:17 edt 2001 i686 unknown ' config_args='-des -Doptimize=-O2 -march=i386 -mcpu=i686 -Dcc=gcc - +Dcccdlflags=-fPIC -Dinstallprefix=/usr -Dprefix=/usr -Darchname=i386- +linux -Dd_dosuid -Dd_semctl_semun -Di_db -Di_ndbm -Di_gdbm -Di_shadow + -Di_syslog -Dman3ext=3pm -Uuselargefiles' hint=recommended, useposix=true, d_sigaction=define usethreads=undef use5005threads=undef useithreads=undef usemultipl +icity=undef useperlio=undef d_sfio=undef uselargefiles=undef use64bitint=undef use64bitall=undef uselongdouble=undef usesocks=u +ndef Compiler: cc='gcc', optimize='-O2 -march=i386 -mcpu=i686', gccversion=2.96 2 +0000731 (Red Hat Linux 7.1 2.96-96) cppflags='-fno-strict-aliasing -I/usr/local/include' ccflags ='-fno-strict-aliasing -I/usr/local/include' stdchar='char', d_stdstdio=define, usevfork=false intsize=4, longsize=4, ptrsize=4, doublesize=8 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=1 +2 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', + lseeksize=4 alignbytes=4, usemymalloc=n, prototype=define Linker and Libraries: ld='gcc', ldflags =' -L/usr/local/lib' libpth=/usr/local/lib /lib /usr/lib libs=-lnsl -ldl -lm -lc -lcrypt libc=/lib/libc-2.2.4.so, so=so, useshrplib=false, libperl=libperl. +a Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-rdynami +c' cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib' Characteristics of this binary (from libperl): Compile-time options: Built under linux Compiled at Aug 9 2001 22:48:52 @INC: /usr/lib/perl5/5.6.0/i386-linux /usr/lib/perl5/5.6.0 /usr/lib/perl5/site_perl/5.6.0/i386-linux /usr/lib/perl5/site_perl/5.6.0 /usr/lib/perl5/site_perl . #Win98 Summary of my perl5 (revision 5 version 6 subversion 1) configuration: Platform: osname=MSWin32, osvers=4.0, archname=MSWin32-x86-multi-thread uname='' config_args='undef' hint=recommended, useposix=true, d_sigaction=undef usethreads=undef use5005threads=undef useithreads=define usemultip +licity=define useperlio=undef d_sfio=undef uselargefiles=undef usesocks=undef use64bitint=undef use64bitall=undef uselongdouble=undef Compiler: cc='cl.exe', ccflags ='-nologo -O1 -MD -DNDEBUG -DWIN32 -D_CONSOLE + -DNO_STRICT -DHAVE_DES_FCRYPT -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLIC +IT_SYS -DPERL_MSVCRT_READFIX', optimize='-O1 -MD -DNDEBUG', cppflags='-DWIN32' ccversion='', gccversion='', gccosandvers='' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234 d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=10 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', + lseeksize=4 alignbytes=8, usemymalloc=n, prototype=define Linker and Libraries: ld='link', ldflags ='-nologo -nodefaultlib -release -libpath:"C:\ +Perl\lib\CORE" -machine:x86' libpth="C:\Perl\lib\CORE" libs= oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib + comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib netap +i32.lib uuid.lib wsock32.lib mpr.lib winmm.lib version.lib odbc32.li +b odbccp32.lib msvcrt.lib perllibs= oldnames.lib kernel32.lib user32.lib gdi32.lib winspool +.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib n +etapi32.lib uuid.lib wsock32.lib mpr.lib winmm.lib version.lib odbc3 +2.lib odbccp32.lib msvcrt.lib libc=C:\Perl\lib\CORE\PerlCRT.lib, so=dll, useshrplib=yes, libperl +=perl56.lib Dynamic Linking: dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' ' cccdlflags=' ', lddlflags='-dll -nologo -nodefaultlib -release -l +ibpath:"C:\Perl\lib\CORE" -machine:x86' Characteristics of this binary (from libperl): Compile-time options: MULTIPLICITY USE_ITHREADS PERL_IMPLICIT_CONTEX +T PERL_IMPLICIT_SYS Locally applied patches: ActivePerl Build 631 Built under MSWin32 Compiled at Jan 2 2002 17:16:22 @INC: C:/Perl/lib C:/Perl/site/lib .
|
|---|