in reply to bug in floating point math?

In a sense, yes, it is a bug. The "known limitation" kind of bug. 1/10 is a periodic number in binary like 1/3 is in decimal. To store it precisely as a float would take infinite storage.

1.1 and 0.1 cut off at a different point, so 1.1-1.0 ends up being slightly different than 0.1.

$ perl -e'printf "%.20e\n", 0.1' 1.00000000000000005551e-01 $ perl -e'printf "%.20e\n", 1.1-1.0' 1.00000000000000088818e-01

Replies are listed 'Best First'.
Re^2: bug in floating point math?
by AnomalousMonk (Archbishop) on Sep 23, 2010 at 04:30 UTC