JSON-XS on quadmath builds of perl

syphilis has asked for the wisdom of the Perl Monks concerning the following question:

Hi,

Internally (in XS,xs), JSON-XS uses Gconvert():

Gconvert (SvNVX (sv), NV_DIG, 0, enc->cur);
[download]

But Gconvert() doesn't DWIM when nvtype is __float128, so I've altered that code to:

#ifdef USE_QUADMATH
quadmath_snprintf(enc->cur, NV_DIG, "%.33Qg", SvNVX (sv)); /* NV_DIG i
+s 33 for quadmath builds */
#else
Gconvert (SvNVX (sv), NV_DIG, 0, enc->cur);
#endif
[download]

I've chosen "33" (== the value of NV_DIG on quadmath builds) because Gconvert uses NV_DIG for both "double" builds of perl (where NV_DIG is 15) and "long double" builds of perl (where NV_DIG is 18).

It seems to work pretty well - but there's one JSON-XS-4.03 test that fails:

t/11_pc_expo.t .......... 1/8
#   Failed test 'digit 1.01e+30'
#   at t/11_pc_expo.t line 35.
#                   '[1010000000000000000000000000000]'
#     doesn't match '(?^:\[1.01[Ee]\+0?30\])'
# Looks like you failed 1 test of 8.
[download]

Before I investigate that test failure, I just want to check that NV_DIG (33) is the correct value to be specifying in the "%Qg" formatting ??
If I alter that value of 33 to 30 (or less), then all tests pass ... but I know very little about json, or what it expects, or what is expected of it.

Any assistance would be much appreciated.

Cheers,
Rob

Comment on JSON-XS on quadmath builds of perl Select or Download Code

Replies are listed 'Best First'.
Re: JSON-XS on quadmath builds of perl by kikuchiyo (Hermit) on Sep 20, 2024 at 08:20 UTC
Hi, I created a fork of JSON::XS called JSON::SIMD that uses the simdjson library for decoding, but retains the encode part with few changes. I also ran into issues with quadmath builds. For the offending line in encode_sv I've used `quadmath_snprintf(enc->cur, enc->end - enc->cur, "%.*Qg", (int)N +V_DIG, SvNVX(sv));` [download] According to my commit message, this fix came from Cpanel::JSON::XS. As for the test, I've "fixed" it like this: `like($js,qr/\[(?:1.01[Ee]\+0?30\|1010000000000000000000000000000)\]/, ' +digit 1.01e+30');` [download] I considered this to be a remote edge case, so I'm not too bothered about it. About the theoretical matters regarding the JSON format... the most recent document closest to a standard is RFC 8259. As you can see it's not very formal, and it says nothing about prescribed ranges in which numbers must be represented as integers or floating point numbers, nor anything about required precision. The remark about interoperability and IEEE 754 doubles indicates that with quadmath number ranges, we are already in "you get to keep the pieces" territory.	[reply] [d/l] [select]
Re^2: JSON-XS on quadmath builds of perl by syphilis (Archbishop) on Sep 20, 2024 at 09:14 UTC
I also ran into issues with quadmath builds. Thanks for the excellent info. In specifying NV_DIG as the second arg handed to quadmath_snprintf() I had made a mistake. It's quite possible that more than 33 characters need to be written to `enc->cur`. I'll switch to `enc->end - enc->cur`. It seems to me that there's no reason to assume that the user wants 33 digits of precision just because perl's nvtype is __float128, but "33" is probably as good an assumption as anything else. Anyone wanting just "double" precision values could alter the "%.33Qg" to "%.15Qg" and rebuild and reinstall the module. (For the same outputs as a "long double" build of perl, you'd specify "%.18Qg".) I think those various options work fine - though I haven't rigorously tested them. Your rewrite of the test looks fine to me, too. The failure of the test is just a quirk of the way perl displays that edge case value: `>perl -le "print 1.01e30;" 1010000000000000000000000000000 <c> Similar things happen with long double values: <c> >perl -le "print 1.01e17;" 101000000000000000` [download] and with doubles, too: `perl -le "print 1.01e14" 101000000000000` [download] Cheers, Rob	[reply] [d/l] [select]
Re^2: JSON-XS on quadmath builds of perl by afoken (Chancellor) on Sep 20, 2024 at 18:30 UTC
About the theoretical matters regarding the JSON format... the most recent document closest to a standard is RFC 8259. As you can see it's not very formal, and it says nothing about prescribed ranges in which numbers must be represented as integers or floating point numbers, nor anything about required precision. I think it does. Basically, any implementation should be able to handle "Numbers" in the range of IEEE 754 doubles, minus NaN and ±Infinity. Especially, the range -2⁵³+1 to 2⁵³-1 should be supported: The representation of numbers is similar to that used in most programming languages. A number is represented in base 10 using decimal digits. It contains an integer component that may be prefixed with an optional minus sign, which may be followed by a fraction part and/or an exponent part. Leading zeros are not allowed. A fraction part is a decimal point followed by one or more digits. An exponent part begins with the letter E in uppercase or lowercase, which may be followed by a plus or minus sign. The E and optional sign are followed by one or more digits. Numeric values that cannot be represented in the grammar below (such as Infinity and NaN) are not permitted. ... This specification allows implementations to set limits on the range and precision of numbers accepted. Since software that implements IEEE 754 binary64 (double precision) numbers IEEE754 is generally available and widely used, good interoperability can be achieved by implementations that expect no more precision or range than these provide, in the sense that implementations will approximate JSON numbers within the expected precision. A JSON number such as 1E400 or 3.141592653589793238462643383279 may indicate potential interoperability problems, since it suggests that the software that created it expects receiving software to have greater capabilities for numeric magnitude and precision than is widely available. Note that when such software is used, numbers that are integers and are in the range [-(253)+1, (253)-1] are interoperable in the sense that implementations will agree exactly on their numeric values. But then again, any implementation is free to set limits on range and precision. So technically, I could limit "Numbers" in a JSON implementation to the range 0.00 to 2.55, with a precision of no more than three digits, and could get away storing "Numbers" in a byte. But I think the spirit of JSON is different. https://www.json.org specifies JSON as being a subset of the JavaScript Programming Language Standard ECMA-262 3rd Edition - December 1999. That document can still be found, and it has a very clear definition of the "Number" type in 4.3.20: The type Number is a set of values representing numbers. In ECMAScript, the set of values represents the double-precision 64-bit format IEEE 754 values including the special “Not-a-Number” (NaN) values, positive infinity, and negative infinity. Alexander -- Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)	[reply]
Re^3: JSON-XS on quadmath builds of perl by syphilis (Archbishop) on Sep 22, 2024 at 06:55 UTC
In ECMAScript, the set of values represents the double-precision 64-bit format IEEE 754 values ... Yes, it seems to me that there would likely be people who want "double-precision 64-bit format IEEE 754 values", irrespective of perl's NVTYPE. However, for long double builds, JSON::XS currently presents "long double" values, so having quadmath builds present "__float128" values is at least consistent with that. I gather that JSON::XS is essentially abandonware (and no-one uses quadmath builds, anyway), but I've placed the patch I've used at https://rt.cpan.org/Ticket/Display.html?id=155704. Thanks kikuchiyo, afoken, for your help. Cheers, Rob	[reply]