Given the pretty CPAN Testers Christmas Tree, especially in the *BSD columns, I learned that not all 32-bit perls are created equal. After some more debug, I was able to show that it was those 32-bit perls with $Config{ivsize}==4 (32bit integers; aka perl -V:ivsize) that were failing, but those with $Config{ivsize}==8 (64bit integers in 32bit perl) would pass.
I had assumed (without looking) that the default ivsize on the 32bit perl was 4 bytes (32 bits), so thought that I had already tested and verified my IV weren't overflowing (or, if they were, they were promoting to NV). After seeing the problem, I discovered that when using left-shift, it would go from IV to NV... but it didn't. However, *=2 did promote the way I expected:
Conclusion: when doing a testing suite across versions, it's not always enough to just have a "32bit perl"; especially if you are dependent on integer sizes, also check that your test suite includes multiple ivsize values.