Re^3: RFC: Large Floating Point Numbers - Rounding Errors (expectations)

I'll take your rant and raise you one...

Flippant disregard for naïveté may be geek-chic, but I live in the real world. That is, while you really do make a valid point, those fourth grade expectations are common, and are consistent in an easily observable way. If I went and told my users that their expectations are merely naive, I would lose.

... shirt, respect, credibility and eventually job.

The best answer noted elsewhere in this thread, is a hardware limitation. The SPARC architecture does not have a way to set the rounding mode for floating point operations. However, x86 Operating Systems (including recent MacOS, but still not Solaris) seems to have a way to reach into the floating point hardware to ask for a NEAR rounding mode. I can sell broken architecture as an excuse (since it is demonstrably true) far better than I can sell broken user expectations (which are not demonstrable at all).

Or, I can cheat, use string_round(), and look like I got a bug fixed where nobody else could do it efficiently. I don't mind cheating, as long as it doesn't cause any real harm. This is such a case, since as BrowserUK dismissively notes above, all rounding is cosmetic anyway.

Comment on Re^3: RFC: Large Floating Point Numbers - Rounding Errors (expectations)

Replies are listed 'Best First'.
Re^4: RFC: Large Floating Point Numbers - Rounding Errors (expectations) by tye (Sage) on Sep 11, 2011 at 06:57 UTC
I'll take your rant and raise you one... Rant? Well, if you want to read it that way, then I can't stop you. The best answer noted elsewhere in this thread, is a hardware limitation. The SPARC architecture does not have a way to set the rounding mode for floating point operations. However, x86 Operating Systems (including recent MacOS, but still not Solaris) seems to have a way to reach into the floating point hardware to ask for a NEAR rounding mode. Um, no. I already replied to that claim showing that a much simpler explanation is that the inconsistent rounding is simply a consequence of converting between base 10 and base 2 and that BrowserUk's results are more due to him computing 0.000035 through repeated additions. I'll be adding a reply shortly that clearly shows that I was correct on that point. The rounding mode makes absolutely no difference in how sprintf rounds 0.000035 and the rounding mode was already at NEAR anyway! Flippant disregard for naïveté may be geek-chic Yeah, again, if you want to read it that way, then I can't stop you. I actually tried to give a useful solution for meeting naive 4th-grade expectations: `printf( "%.5f\n", sprintf("%.12e",$_) )` [download] (except I left off inserting the '1' by mistake, which would complicate the code a bit, but nowhere near as much as your "nobody else could do it efficiently" code.) I decided not to go into it in much detail such as noting that the value for 12 can be somewhat different on different builds. I'd hoped Config exposed the number of digits Perl considered significant but it doesn't seem to. Replace the 12 with a value perhaps equal or slightly less than Perl's "significant digits" count. - tye	[reply] [d/l]

Replies are listed 'Best First'.

Re^4: RFC: Large Floating Point Numbers - Rounding Errors (expectations)
by tye (Sage) on Sep 11, 2011 at 06:57 UTC

I'll take your rant and raise you one...

Rant? Well, if you want to read it that way, then I can't stop you.

The best answer noted elsewhere in this thread, is a hardware limitation. The SPARC architecture does not have a way to set the rounding mode for floating point operations. However, x86 Operating Systems (including recent MacOS, but still not Solaris) seems to have a way to reach into the floating point hardware to ask for a NEAR rounding mode.

Um, no. I already replied to that claim showing that a much simpler explanation is that the inconsistent rounding is simply a consequence of converting between base 10 and base 2 and that BrowserUk's results are more due to him computing 0.000035 through repeated additions.

I'll be adding a reply shortly that clearly shows that I was correct on that point. The rounding mode makes absolutely no difference in how sprintf rounds 0.000035 and the rounding mode was already at NEAR anyway!

Flippant disregard for naïveté may be geek-chic

Yeah, again, if you want to read it that way, then I can't stop you. I actually tried to give a useful solution for meeting naive 4th-grade expectations:

printf( "%.5f\n", sprintf("%.12e",$_) )
[download]

(except I left off inserting the '1' by mistake, which would complicate the code a bit, but nowhere near as much as your "nobody else could do it efficiently" code.) I decided not to go into it in much detail such as noting that the value for 12 can be somewhat different on different builds. I'd hoped Config exposed the number of digits Perl considered significant but it doesn't seem to. Replace the 12 with a value perhaps equal or slightly less than Perl's "significant digits" count.

- tye

[reply]
[d/l]