cross posted to p5p mailing list
I decided to test a blead perl compiled with Visual C 2003 against Visual C 2008 with perlbench-run. With ithreads and psuedofork (normal win32 config) on 32 bit windows. The 2 tests were run in a Perl folder, that was identical except for perl517.dll and HiRes.dll (the VC 2003 HiRes.dll crashed when run with a VC 2008 perl517.dll, I did not pursue the bug) which were copied from the VC 2008 build on another machine. So all XS dlls on the VC 2008 were compiled with VC 2003 except for Time::HiRes. explorer.exe was the only GUI process running during the test with a context switch delta above 2 in Process Explorer so CPU had almost no background load, I killed every thing else. I prune the Win32 services list to a minimum on my PCs. Tests were on a T7200 Core 2 Duo with CPUs locked to 2 GHZ (no power savings). perlbench-run is not portable and is not Windows compatible. I did a bare minimum of changes to it for it to run without fatally erroring on Windows. -G7 flag does not exist on 2008. I saw no replacement for it on 2008. Remember the 2003 and 2008 perl517s are linked against different CRTs/clibs. Security cookie aka stack frame overrun checking (-GS) is off by default in 2003, I turned it off in 2008 for apple to apple (not standard in Perl's win32 makefile). I also turned on -GL (inter procedure optimization) for 2003 and 2008 to give each the best chance to optimize (not standard in Perl's win32 makefile).
VC 2003 optimize='-MD -Zi -DNDEBUG -O1 -G7 -GL -arch:SSE2'
VC 2008 optimize='-MD -Zi -DNDEBUG -O1 -GL -GS- -arch:SSE2'
Not Cfg=Debug.
Both are "Snapshot of: 5c26a17602c4585ca111bd154f6c65c681cd0572"
VC 2003 perl517.dll 956 KB on disk
.text 0xC17CF
.rdata 0x2140A
.data 0x3938
.reloc 0x5F38
VC 2008 perl517.dll 975 KB on disk
.text 0xC834A
.rdata 0x2178A
.data 0x3C58
.rsrc 0x3C0
.reloc 0x622C
In trying to figure out why the code bloat, the most striking thing was, for sv->sv_flags, VC 2003 tests them with
test char * [sv+13], 0x08, while VC 2008 does
test I32 * [sv+12], 0x00000800 which takes much more machine code to fit that constant in. There are probably other things my eye didn't notice. Anyway, here the results.
First perlbench run, it took about 5 mins to run.
C:\sources\perlbench-0.93>perl perlbench-run -c 100000 "c:/perl517/bin
+/perl.exe"
"C:\perl517\vc08\bin\perl.exe"
Use of uninitialized value in concatenation (.) or string at perlbench
+-run line
75.
A)
version = 5.017006
path = c:/perl517/bin/perl.exe
Use of uninitialized value in concatenation (.) or string at perlbench
+-run line
75.
B)
version = 5.017006
path = C:\perl517\vc08\bin\perl.exe
A B
--- ---
arith/mixed 100 114
arith/trig 100 121
array/copy 100 103
array/foreach 100 95
array/index 100 102
array/pop 100 103
array/shift 100 104
array/sort-num 100 100
array/sort 100 110
call/0arg 100 104
call/1arg 100 97
call/2arg 100 105
call/9arg 100 97
call/empty 100 96
call/fib 100 91
call/method 100 90
call/wantarray 100 108
hash/copy 100 100
hash/each 100 105
hash/foreach-sort 100 103
hash/foreach 100 96
hash/get 100 105
hash/set 100 105
loop/for-c 100 106
loop/for-range-const 100 88
loop/for-range 100 89
loop/getline 100 103
loop/while-my 100 102
loop/while 100 101
re/const 100 99
re/w 100 97
startup/fewmod 100 -
startup/lotsofsub - -
startup/noprog 100 86
string/base64 100 99
string/htmlparser 100 101
string/index-const 100 121
string/index-var 100 102
string/ipol 100 111
string/tr 100 94
AVERAGE 100 101Use of uninitialized value $str in
+ substitut
ion (s///) at perlbench-run line 375.
Use of uninitialized value $str in substitution (s///) at perlbench-ru
+n line 376
.
Use of uninitialized value in concatenation (.) or string at perlbench
+-run line
287.
Use of uninitialized value $str in substitution (s///) at perlbench-ru
+n line 375
.
Use of uninitialized value $str in substitution (s///) at perlbench-ru
+n line 376
.
Use of uninitialized value in concatenation (.) or string at perlbench
+-run line
287.
Results saved in file:///C|/sources/perlbench-0.93/benchres-001/index.
+html
C:\sources\perlbench-0.93>
second run of perlbench, i upped the try count to see if anything signifigant shows up at higher test cycles. It took about 30 mins.
C:\sources\perlbench-0.93>perl perlbench-run -c 500000 "c:/perl517/bin
+/perl.exe"
"C:\perl517\vc08\bin\perl.exe"
Use of uninitialized value in concatenation (.) or string at perlbench
+-run line
75.
A)
version = 5.017006
path = c:/perl517/bin/perl.exe
Use of uninitialized value in concatenation (.) or string at perlbench
+-run line
75.
B)
version = 5.017006
path = C:\perl517\vc08\bin\perl.exe
A B
--- ---
arith/mixed 100 107
arith/trig 100 120
array/copy 100 103
array/foreach 100 95
array/index 100 99
array/pop 100 104
array/shift 100 104
array/sort-num 100 99
array/sort 100 109
call/0arg 100 106
call/1arg 100 96
call/2arg 100 104
call/9arg 100 98
call/empty 100 97
call/fib 100 92
call/method 100 91
call/wantarray 100 105
hash/copy 100 100
hash/each 100 104
hash/foreach-sort 100 103
hash/foreach 100 97
hash/get 100 103
hash/set 100 108
loop/for-c 100 104
loop/for-range-const 100 90
loop/for-range 100 89
loop/getline 100 104
loop/while-my 100 103
loop/while 100 102
re/const 100 99
re/w 100 99
startup/fewmod 100 90
startup/lotsofsub 100 -
startup/noprog 100 104
string/base64 100 100
string/htmlparser 100 101
string/index-const 100 118
string/index-var 100 101
string/ipol 100 110
string/tr 100 91
AVERAGE 100 101Use of uninitialized value $str in
+ substitut
ion (s///) at perlbench-run line 375.
Use of uninitialized value $str in substitution (s///) at perlbench-ru
+n line 376
.
Use of uninitialized value in concatenation (.) or string at perlbench
+-run line
287.
Use of uninitialized value $str in substitution (s///) at perlbench-ru
+n line 375
.
Use of uninitialized value $str in substitution (s///) at perlbench-ru
+n line 376
.
Use of uninitialized value in concatenation (.) or string at perlbench
+-run line
287.
Results saved in file:///C|/sources/perlbench-0.93/benchres-002/index.
+html
C:\sources\perlbench-0.93>
So what is the lesson here? Older is better? newer is not better? Or useless mucking by MS over the years to the compiler that turns out to cancel itself out? Or am I testing the frequency of hardware interrupts and ethernet broadcast traffic? or "faster" and "aligned" is self defeating if it causes something to not be in the CPU cache because it is fatter? Or am I testing changes of MS's clibs from 2003 to 2008? or is perlbench-run a bogus inaccurate test that does not reflect real world conditions?