in reply to RFC: Large Floating Point Numbers - Rounding Errors

This method fixes 0.000035 rounding down issue.

What issue do you have with 0.000035 rounding to 0.00003 when you've explicitly asked for 5 dp?

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
  • Comment on Re: RFC: Large Floating Point Numbers - Rounding Errors

Replies are listed 'Best First'.
Re^2: RFC: Large Floating Point Numbers - Rounding Errors
by GAVollink (Novice) on Sep 08, 2011 at 14:42 UTC
    It is not consistent:

    0.000005 up 0.00001
    0.000015 up 0.00002
    0.000025 up 0.00003
    0.000035 dn 0.00003
    0.000045 up 0.00005
    0.000055 up 0.00006
    0.000065 dn 0.00006
    0.000075 dn 0.00007
    0.000085 up 0.00009
    0.000095 up 0.00010
[reply]

      Very strange. I'd almost say that you'd discovered a bug in perl or the underlying C-libraries that was causing the double precision (64-bit) NV value to be transitioned through a single precision (32-bit) float whilst being formatted.

      If the values were always being manipulated as doubles, this shouldn't happen.

      It could be a that the IEEE rounding mode isn't being set (correctly?).

      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
        This is not a bug but the result of converting numbers between base 10 and 2.

        The problem is that 0.000005, 0.000015, 0.000025, 0.000035, 0.000045, 0.000055, 0.000065, ... can not be represented precisely in base 2: 

        printf("%.5f %.30f\n", $_, $_) for (0.000005, 0.000015, 0.000025, 0.00
+0035, 0.000045, 0.000055, 0.000065)

        [download]
[reply]
[d/l]

In Section Meditations