semuel has asked for the wisdom of the Perl Monks concerning the following question:

Hi All.

I am the owner of a module called Data::ParseBinary, and recently I am adding support for big integers. (64 bit)
However, some of the tests fail on systems that have native 64 bit compiled in.

Can someone with tell me why this code: (you will need Perl 5.10 and 64 bit support and Math::BigIn installed)

print unpack "H*", pack "Q>", Math::BigInt->new("18446744073709551360" +);

prints 'ffffffffffffffff' instead of 'ffffffffffffff00' that I expect it to?
(those with Perl 5.8 can replace "Q>" with "Q")

I also see reports from Perl 5.6.2 with 64 bit support. I get a weird result there with all-zeros.

Help?

Replies are listed 'Best First'.
Re: 64 bit integers woes
by ikegami (Patriarch) on Sep 07, 2009 at 05:56 UTC
    I don't get that behaviour on the 64-bit 5.8.8 Perl (little-endian system) to which I have access.
    $ perl -MMath::BigInt -le'print unpack "H*", pack "Q", Math::BigInt->n +ew("18446744073709551360");' 00ffffffffffffff $ perl -MMath::BigInt -le'print unpack "H*", "".reverse pack "Q", Math +::BigInt->new("18446744073709551360");' ffffffffffffff00

      You are right, after farther inspection I see that some of the test pass on 64 bit enabled.

      still, I can't figure out why on some 64 bit systems it give the correct result, and on other 64 bit systems it fails.

      And on the 5.6 front, every 64 bit system fails.
        Does it fail if you explicitly ->numify the BigInt?
Re: 64 bit integers woes
by syphilis (Archbishop) on Sep 07, 2009 at 09:45 UTC
    It's a bit weird.
    Firstly, I'm getting the same as you with perls built with -Duse64bitint - except that it works as expected *iff* that perl has also been built with -Duselongdouble. (I don't know why this is so, and I haven't found time to have a closer look.)
    Secondly (and you probably already know this), the problem only arises with Math::BigInt objects. A normal perl scalar containing the value 18446744073709551360 is fine. Why are you storing this value in a Math::BigInt object, given that a perl scalar can store that value ?

    Cheers,
    Rob
      In 32-bit Perls, the integer type (IV) has 32-bit of precision, while the floating-point type (NV) has 53-bit of precision. As a result, some places assume that the NV can hold larger numbers than IVs. I guess the OP found the effects of one.

      Math::BigInt(::FastCalc) numifies values as an NV:

      void _num(class,x) SV* x INIT: AV* a; NV fac; SV* temp; NV num; I32 elems; I32 index; NV BASE; CODE: a = (AV*)SvRV(x); /* ref to aray, don't check ref */ elems = av_len(a); /* number of elems in array */ if (elems == 0) /* only one element? */ { ST(0) = *av_fetch(a, 0, 0); /* fetch first (only) element */ XSRETURN(1); /* return it */ } fac = 1.0; /* factor */ index = 0; num = 0.0; BASE = XS_BASE; while (index <= elems) { temp = *av_fetch(a, index, 0); /* fetch current element */ num += fac * SvNV(temp); fac *= BASE; index++; } ST(0) = newSVnv(num);

      pack'Q' extracts the value to be packed from the SV using SvUV:

      #ifdef HAS_QUAD case 'Q': while (len-- > 0) { Uquad_t auquad; fromstr = NEXTFROM; auquad = (Uquad_t) SvUV(fromstr); #### here! DO_BO_PACK(auquad, 64); PUSH_VAR(utf8, cur, auquad); }

      SvUV() silently mangles NVs > 2**53:

      printf "%3d %20d %s\n", $_, ( 1 << $_ )-1, unpack 'h*', pack 'Q>', Math::BigInt->new( ( 1 << $_ )-1 ) for 50 .. 60;; 50 1125899906842623 0030ffffffffffff 51 2251799813685247 0070ffffffffffff 52 4503599627370495 00f0ffffffffffff 53 9007199254740991 00f1ffffffffffff 54 18014398509481983 0004000000000000 55 36028797018963967 0008000000000000 56 72057594037927935 1000000000000000 57 144115188075855871 2000000000000000 58 288230376151711743 4000000000000000 59 576460752303423487 8000000000000000 60 1152921504606846975 0100000000000000

      After that it gets a little fuzzy. SvUV() appears to be implemented as Perl_sv_2uv_flags(), and if SvNOKp is set (which it would be), then the coersion is done by the macro U_V:

      if (SvNOKp(sv)) return U_V(SvNVX(sv));

      which is defined in perl.h. There is code to detect out-of-range FP values, but whther it is compiled in or not is dependant upon another #define:

      /* This may look like unnecessary jumping through hoops, but convertin +g out of range floating point values to integers *is* undefined behav +iour, and it is starting to bite. */ #ifndef CAST_INLINE #define I_32(what) (cast_i32((NV)(what))) #define U_32(what) (cast_ulong((NV)(what))) #define I_V(what) (cast_iv((NV)(what))) #define U_V(what) (cast_uv((NV)(what))) #else #define I_32(n) ((n) < I32_MAX_P1 ? ((n) < I32_MIN ? I32_MIN : (I32) ( +n)) \ : ((n) < U32_MAX_P1 ? (I32)(U32) (n) \ : ((n) > 0 ? (I32) U32_MAX : 0 /* NaN */))) #define U_32(n) ((n) < 0.0 ? ((n) < I32_MIN ? (UV) I32_MIN : (U32)(I32 +) (n)) \ : ((n) < U32_MAX_P1 ? (U32) (n) \ : ((n) > 0 ? U32_MAX : 0 /* NaN */))) #define I_V(n) ((n) < IV_MAX_P1 ? ((n) < IV_MIN ? IV_MIN : (IV) (n)) \ : ((n) < UV_MAX_P1 ? (IV)(UV) (n) \ : ((n) > 0 ? (IV)UV_MAX : 0 /* NaN */))) #define U_V(n) ((n) < 0.0 ? ((n) < IV_MIN ? (UV) IV_MIN : (UV)(IV) (n) +) \ : ((n) < UV_MAX_P1 ? (UV) (n) \ : ((n) > 0 ? UV_MAX : 0 /* NaN */))) #endif

      which apparently isn't being set in 64-bit builds?


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.

        Math::BigInt(::FastCalc) numifies values as an NV:

        That would account for a loss of precision, but it shouldn't result in -1.

        SvUV() silently mangles NVs > 2**53

        Are you sure? The number the OP posted requires 56 bits of precision to be stored in a float (64 bits for an integer). One would think the value was already mangled.

        but whther it is compiled in or not is dependant upon another #define

        CAST_INLINE only determines if the cast is done inline or if a subroutine is called to do the cast. Or where you referring to some unshown #ifdef?

      thanks for the clue. I'll try to look up the source of things.

      The code in question is part of an unit test.
      the module itself should know if the environment support 64 bit, and return (in the relevant function) either Math::BigInt object or simple scalar, if the system support 64 bit or not.

      However, the test is written to not care. so I wrote the test using Math::BigInt, and thought that it will handle all the cases. apparently not.

Re: 64 bit integers woes
by DrHyde (Prior) on Sep 07, 2009 at 09:46 UTC
    If you contact the CPAN-testers who are reporting failures, then maybe one of them can give you access to a machine on which you can try stuff out.