Help with x86_64 bit testing for CPAN module

Background: The other day hangon was looking for a way of Randomizing Large Integers. BrowserUk soundly suggested Math::Random::MT for the task. Although this is a perfectly good solution it was not viable due to restriction on compiled code. Having nothing better to do for a couple of hours I had a look at Math::Random::MT. After patching the header file so it would install on Win32 (should fix the compilation failures on Darwin too) I noted it would be easy enough to port to vanilla perl, and thus Math::Random::MT::Perl was born.

Like the other ::Perl modules it exists only to plug the gap when you can't install better/faster C/XS. Anyway after a bit of mucking around this pure Perl version of the Mersene Twister PRNG now exists on CPAN. On the 32 bit platforms I have access to it works fine (producing identical output to the C at an acceptable 100k PRNs/sec) however it breaks on 64 bit architecture. This occurs due to the need to 'use integer' in parts to get the correct integer wrap behaviour, but on 64 bit machines the wrap does not happen at 2**32-1, it happens at 2**64-1. As you can't directly specify a uint32_t within perl this presents a problem.

I think a patch like this:

{ 
    use integer; 
    $int = $int * $big_num;             # need 32 bit int wrap here bu
+t don't get on 64 bit arch
    $int >>= 32 if $int > 0xffffffff;  # <-- add this line
}
[download]

should fix the issue by constraining the 64 bit ints to 32 bits, effectively forcing the wrap. I can't test this as I don't have access to a 64 bit perl. While I know this won't break things on 32 bit arch I don't really want to upload it just to see if it works when a 64 bit tester runs it. There is modified module here. If anyone with a 64 bit perl has a moment could you run:

wget http://74.55.157.146/Math-Random-MT-Perl-1.02.tar.gz && \
tar -xzf Math-Random-MT-Perl-1.02.tar.gz && \
cd Math-Random-MT-Perl-1.02 && \
perl Makefile.PL && make test
[download]

With any luck it should come up with "All tests successful". If not the output would be helpful.

If it does run OK and you have a couple of minutes there is a little script in /t called validate.pl. You would need to actually install both Math::Random::MT and Math::Random::MT::Perl. A quick run of this does a lot more in depth testing to ensure that the C and Perl results are identical. A command line arg of 1000 will do 2 million comparison tests and should be over in under 20 seconds.

Comment on Help with x86_64 bit testing for CPAN module Select or Download Code

Replies are listed 'Best First'.
Re: Help with x86_64 bit testing for CPAN module by almut (Canon) on May 22, 2008 at 13:27 UTC
Haven't yet looked into why it's failing... but for the moment, here's the output with a "perl, v5.8.8 built for x86_64-linux-thread-multi": PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_h +arness(0, 'blib/lib', 'blib/arch')" t/.t t/1....ok 1/4# Test 3 got: "0.814723691903055" (t/1.t at line 13) + t/1....NOK 3# Expected: "0.00900305295363069" + # t/1.t line 13 is: ok(0.814723691903055, $gen->rand()); # Test 4 got: "0.135477004107088" (t/1.t at line 14) t/1....NOK 4# Expected: "0.592605744022876" + # t/1.t line 14 is: ok(0.135477004107088, $gen->rand()); t/1....FAILED tests 3-4 + Failed 2/4 tests, 50.00% okay t/2....ok + t/3....ok + t/4....ok 1/3# Test 2 got: "0.814723691903055" (t/4.t at line 16) + # Expected: "0.00900305295363069" # t/4.t line 16 is: ok(0.814723691903055, rand()); # Test 3 got: "0.135477004107088" (t/4.t at line 17) # Expected: "0.592605744022876" # t/4.t line 17 is: ok(0.135477004107088, rand()); t/4....FAILED tests 2-3 + Failed 2/3 tests, 33.33% okay t/5....ok 1/5# Test 3 got: "0.114662232343107" (t/5.t at line 17) + t/5....NOK 3# Expected: "0.67886575916782" + # t/5.t line 17 is: ok( $gen->rand(1), 0.67886575916782 ); # Test 5 got: "0.621264568995684" (t/5.t at line 21) t/5....NOK 5# Expected: "0.336814725538716" + # t/5.t line 21 is: ok( $gen->rand(1), 0.336814725538716 ); t/5....FAILED tests 3, 5 + Failed 2/5 tests, 60.00% okay Failed Test Stat Wstat Total Fail Failed List of Failed ---------------------------------------------------------------------- +--------- t/1.t 4 2 50.00% 3-4 t/4.t 3 2 66.67% 2-3 t/5.t 5 2 40.00% 3 5 Failed 3/5 test scripts, 40.00% okay. 6/26 subtests failed, 76.92% oka +y. make: ** [test_dynamic] Error 255 [download]	[reply] [d/l]
Re^2: Help with x86_64 bit testing for CPAN module by tachyon-II (Chaplain) on May 22, 2008 at 14:19 UTC
Hmm, thanks for that. I am using the tests out of the C module and unfortunately they got the order wrong so you need to reverse got/expected. Before the patch it was failing like this: `t/1......# Test 3 got: "0.814723691903055" (t/1.t at line 13) # Expected: "873847195.017769552" # t/1.t line 13 is: ok(0.814723691903055, $gen->rand()); # Test 4 got: "0.135477004107088" (t/1.t at line 14) # Expected: "2307073947.03889426" [snip]` [download] In the first test it expects 0.814xxxxx and was getting a large float becuase the final stage on output is to multiply the "uint32" by (1.0/4294967296.0). If the uint32 is actually a uint64 you get a big float. Obviously the patch is now appropriately constraining the int to 32 bits but there is still an issue. I would not ask you to debug it but if you were going to look the algorithm consists of a setup phase mt_init_seed() and an extraction phase mt_genrand(). Only failures in 5.t relate to mt_setup_array() so that code is not relevant. `Adding a debugging printf to the C here: void mt_init_seed( struct mt m, uint32_t seed ) { int i; uint32_t mt; mt = m->mt; mt[0] = seed & 0xffffffff; for ( i = 1; i < N; i++ ) { mt[i] = 1812433253 * (mt[i-1]^(mt[i-1]>>30)) + i; printf("init %d: %u\n", i, mt[i]); // what we want } m->mti = N; } and running $ perl -MMath::Random::MT -e "Math::Random::MT->new(1)" > out` [download] Gives the setup for a seed of value 1 (included after readmore) If you add that line to the equivilent place in the Perl version (around line 48): `sub mt_init_seed { my ($self, $seed) = @_; my @mt; $mt[0] = $seed & 0xffffffff; for ( my $i = 1; $i < $N; $i++ ) { my $xor = $mt[$i-1]^($mt[$i-1]>>30); { use integer; $mt[$i] = 1812433253 * $xor + $i; $mt[$i] >>= +32 if $mt[$i] > 0xffffffff } printf("init %d: %u\n", $i, $mt[$i]); } $self->{mt} = \@mt; $self->{mti} = $N; } and running perl -MMath::Random::MT::Perl -e "Math::Random::MT::Perl->new(1)" > ou +tp` [download] Would let me see if the setup is working as expected. Here is the expected output for a seed value 1 from the C. Read more... (13 kB)	[reply] [d/l] [select]
Re^3: Help with x86_64 bit testing for CPAN module by almut (Canon) on May 22, 2008 at 14:55 UTC
`perl -MMath::Random::MT::Perl -e "Math::Random::MT::Perl->new(1)" > ou +tp` [download] Would let me see if the setup is working as expected... I get a more 'sawtooth'-like pattern like this: init 1: 1812433254 init 2: 764828710 init 3: 322750068 init 4: 136197301 init 5: 57473899 init 6: 24253410 init 7: 10234696 init 8: 4318939 init 9: 1822549 init 10: 769097 init 11: 324551 init 12: 136957 init 13: 57794 init 14: 24388 init 15: 10291 init 16: 4342 init 17: 1832 init 18: 773 init 19: 326 init 20: 137 init 21: 57 init 22: 24 init 23: 10 init 24: 4 init 25: 1 init 26: 1812433279 init 27: 764828720 init 28: 322750072 init 29: 136197303 init 30: 57473900 init 31: 24253411 init 32: 10234696 init 33: 4318939 init 34: 1822549 init 35: 769097 init 36: 324551 init 37: 136957 init 38: 57794 init 39: 24388 init 40: 10291 init 41: 4342 init 42: 1832 init 43: 773 init 44: 326 init 45: 137 init 46: 57 init 47: 24 init 48: 10 init 49: 4 init 50: 1 ... [download] (of course, I could also send you the whole output if it helps...) BTW, here's the respective line from the `perl -V` output `use64bitint=define use64bitall=define uselongdouble=undef` [download] which looks to me like 64-bit ints have in fact been configured...	[reply] [d/l] [select]
Re: Help with x86_64 bit testing for CPAN module by almut (Canon) on May 22, 2008 at 18:23 UTC
`my $xor = $mt[$i-1]^($mt[$i-1]>>30); { use integer; $mt[$i] = 1812433253 * $xor + $i; $mt[$i] >>= +32 if $mt[$i] > 0xffffffff }` [download] It seems to work (at least this part), if you replace `$mt[$i] >>= 32 if $mt[$i] > 0xffffffff;` [download] with `$mt[$i] &= 0xffffffff;` [download] in `mt_init_seed()`, i.e. simply truncate the higher bits. In any case I then get the same init number sequence as with the `Math::Random::MT` C version. Update: and if I replace the other 3 occurrences of the same pattern in `mt_setup_array()`, too, it does in fact pass all five tests...	[reply] [d/l] [select]
Re^2: Help with x86_64 bit testing for CPAN module by tachyon-II (Chaplain) on May 22, 2008 at 23:59 UTC
Doh. How stupid do I feel now you point out the blindingly obvious! Of course we just want the low order bits for the wrap as you point out. To make it even worse there is already and seed & 0xffffffff immdediately above and this was my first thought of how to do it. I rejected it due to temporary insanity. Many thanks. The sawtooth pattern was pretty, and predictable given what I was doing. Cheers James	[reply]
Re: Help with x86_64 bit testing for CPAN module by moritz (Cardinal) on May 22, 2008 at 12:59 UTC
I have no 64 bit perl available atm, but perhaps you'd still like to know that I tested it successfully on two Debian GNU/Linux version, on "Etch" aka stable with perl 5.8.8 and "Lenny" aka testing with perl 5.10.0.	[reply]
Re: Help with x86_64 bit testing for CPAN module by DrHyde (Prior) on May 28, 2008 at 10:27 UTC
I don't really want to upload it just to see if it works when a 64 bit tester runs it Why not? Just upload a development version - ie one with a _ in the version number - and testers will test it but it won't be indexed, so ordinary users won't ever see it. Speaking as a tester, I certainly wouldn't object to you doing this.	[reply]
Re^2: Help with x86_64 bit testing for CPAN module by tachyon-II (Chaplain) on May 28, 2008 at 16:39 UTC
Fair enough. Jobs done as far as portability goes. Now just need to get enough tuits to fix up the Kwalitee gripes.....	[reply]