in reply to Re^4: Numeric Comparisons Randomly Giving the Wrong Result
in thread Numeric Comparisons Randomly Giving the Wrong Result

Sure, but the point is that altho you see a number ending with ".33333" printed out, it may not have that exact value in memory, and if further calculations are performed, that could become apparent. Similarly, of course, decimal numbers cannot truly represent the fraction 1/3. The "galactic credits" example is a bit disingenuous -- when was the last time you wrote a check for $833.3333 to pay your mortgage? Again, money has a fixed precision of two decimal places and needs to be kept there. If rounding is an issue, use tenths of a cent as an integer unit and write a rounding function. This could be as simple as >=/< 5, since integers are round down (so what would have been 4.9999999 tenths of a cent will be 4, anything more will be five).

If you think it is more appropriate to round up (the bank wants at least what it is owned, ie, $8.34) you should still limit the precision with integers, otherwise you will end up with numbers like 1000.00000001 due to the floating point issues that motivated the original post, and if those are rounded up and collated you could be looking at whole dollars of error.

  • Comment on Re^5: Numeric Comparisons Randomly Giving the Wrong Result

Replies are listed 'Best First'.
Re^6: Numeric Comparisons Randomly Giving the Wrong Result
by mjscott2702 (Pilgrim) on Oct 01, 2010 at 10:59 UTC
    OK, so the example was a bit artificial. But the point still holds, especially for a financial/scientific application.

    Ever watch Brewster's Millions?