IEEE double-precision float is 53 bit float. 0.00000000002910383046 = 2**-35. With the 18 bits for the integral portion of 255160 or 256160, that's the full 53bits. So, apparently, when you take the 53bit fractional representation of 255.16 multiplied by 1000dec, round-off sets the final bit to 0; when you take 256.16 * 1000, round-off sets the final bit to 1. Now (when I have time) I'm going to have to figure out the formula to figure out which powers of two will have this error and which won't. It all depends on the 53rd bit of the product of 2**N+16/25 times 128 (same as the bits of times 1000, since the factor of 8 is a binary shift). Oh, I'm nearly there... the 125 is 0b111_1101... and that will just interact with the 53rd bit... urgh but I've got to leave now... I guess it's an exercise for later. :-)

In reply to Re^3: Integers sometimes turn into Reals after substraction (error correction ?) by pryrt
in thread Integers sometimes turn into Reals after substraction by rduke15

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.