in reply to Re^7: test fails on 64bit uselongdouble Perl
in thread test fails on 64bit uselongdouble Perl

The result needs to be an integer but the calculation includes floating-point numbers. Again, my problem is that I'm getting different results on 64bit uselongdouble Perl and may best theory currently is differences in floating-point math on different Perls.

  • Comment on Re^8: test fails on 64bit uselongdouble Perl

Replies are listed 'Best First'.
Re^9: test fails on 64bit uselongdouble Perl
by moritz (Cardinal) on Oct 29, 2009 at 22:22 UTC
    Yes, I understood that. And yes, there are differences in floating point handling.

    But I still don't get what's wrong with rounding to an integer - care to explain?

    You currently seem to think that an integral result as an outcome is normal, but as far as I can tell it works only accidentally on some perls, even if that's the majority of perls.

    Perl 6 - links to (nearly) everything that is Perl 6.

      Ahhh, I see what you're saying and you make a good point.

      What I'm doing is giving my algorithm a latitude and longitude and it's combining it with some other values it has about a weather data file and it comes back with in integer that's an index into a binary structure of 32bit weather data. My problem comes when I give it a lat/long pair (which may include fractions of a degree), that is exactly one of the pairs that has a data item, I want to get the right index. And I don't always get the right index on all Perls.

      For other lat/long pairs that are between data values rounding is my only option so you're right about that. My algorithm does in fact always round and "accidentally" works on most Perls

      My "problems" started recently when I added support for a new type of file that happened to have several hundred duplicate values. It was a projection to a grid with 180 degree lines on both the east and west side. This let me write a test that asserted that these should be the same value and just one out of 301 failed on uselongdouble. And now here we are ;-)

      (BTW, writing this just gave me an idea which I will post latter... )

        My algorithm does in fact always round

        As I mentioned a couple of times already (1, 2), you showed it to be truncating (not rounding).