in reply to Re: OT: How much float precision needed for operation?
in thread OT: How much float precision needed for operation?

So, I think what your data says is:
To ensure an error of no more than plus/minus 0.1 we'd need ~7 digits to the right of the decimal point for y=10 and ~53 digits for y=100
Is this correct?

-Scott

Update: trammel, thank you very much for your excellent examples above.

So, for others who want to extend this to other functions:

If we take the derivative of the function we're working with, multiplied by a delta (difference), and know what our tolerance for error is, we can solve for the maximum delta that will be within our tolerance for error.

In math symbols, the process is:

  • Comment on Re^2: OT: How much float precision needed for operation?