Re^6: Reading (the same) data in different ways & memory usage

I would expect this to be the case for Fixed-Point values (DECIMAL and NUMERIC types), but not for FLOAT and DOUBLE, which already suffer from floating point representation errors because, well, that's what they are (even inside MySQL).

Comment on Re^6: Reading (the same) data in different ways & memory usage

Replies are listed 'Best First'.
Re^7: Reading (the same) data in different ways & memory usage by BrowserUk (Patriarch) on Apr 21, 2011 at 09:13 UTC
They may be stored on disk as IEEE floats (or some other binary representation), but if they are defined as FLOAT(7,2), then the DB needs to return them to you in that form. Ie. with no more than 2 decimal places. But if the value arises due to math internal to the DB, then the binary representation may contain more than two decimal places. just as when you do math in perl. Eg. `$n = 1/10;; printf "%.17f\n", $n;; 0.10000000000000001` [download] So, when it returns a value it needs to round or truncate to meet the column type, and the easiest way to do that is sprintf which returns a string. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply] [d/l]

Replies are listed 'Best First'.

Re^7: Reading (the same) data in different ways & memory usage
by BrowserUk (Patriarch) on Apr 21, 2011 at 09:13 UTC

They may be stored on disk as IEEE floats (or some other binary representation), but if they are defined as FLOAT(7,2), then the DB needs to return them to you in that form. Ie. with no more than 2 decimal places.

But if the value arises due to math internal to the DB, then the binary representation may contain more than two decimal places. just as when you do math in perl. Eg.

$n = 1/10;;
printf "%.17f\n", $n;;
0.10000000000000001
[download]

So, when it returns a value it needs to round or truncate to meet the column type, and the easiest way to do that is sprintf which returns a string.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

[reply]
[d/l]