pimperator has asked for the wisdom of the Perl Monks concerning the following question:

I have a file that's ~2.3 million lines long and has three columns I'm only interested in the third column. I need to add up all of the variables. First I thought it was due to scientific notation (-0.2E-4). But a made copies of the raw files in decimal notation, and that did not work. Could it be that it's too large for perl? or it could be a non-numeric variable in the data file, so how would I test for that?

Replies are listed 'Best First'.
Re: Perl returns NaN after addition
by LanX (Saint) on Oct 17, 2013 at 00:45 UTC
    Thanks to my good relation to the NSA I was able to break into your computer and look at the data you are hiding from us. :)

    It looks like you had a non-numeric variable in the data file!

    > so how would I test for that?

    stop reading the file as soon as you get NaN. Print the line number '$.' and dump line and column with Data::Dumper

    BTW: did you use warnings?

    Cheers Rolf

    ( addicted to the Perl Programming Language)

[reply]
[d/l]
      What is the regex for testing if a variable is NaN? and warnings did not spit out anything. thanks!
[reply]
        from FAQ

        Scalar::Util (distributed with 5.8) provides access to perl's internal function looks_like_number for determining whether a variable looks like a number.

        update

        despite older discussions recommending that way I do have problems on my system:

          DB<243> $nan=sin(9**9**9)
 => "nan"

  DB<244>  looks_like_number($nan)
 => 8704

  DB<245> $nan eq "nan"
 => 1

  DB<246>  Scalar::Util::looks_like_number($nan)
 => 36

        [download]

        I know that NaN is plattform/C-compiler depended, so better test which way works.

        update

        according to perlop#Equality Operators 

          DB<106> $nan=sin(9**9**9)
 => "nan"

  DB<107> print "NaN support here" if $nan != $nan
 => 1
NaN support here
  DB<108>

        [download]

        Cheers Rolf

        ( addicted to the Perl Programming Language)

[reply]
[d/l]
[select]
Re: Perl returns NaN after addition
by syphilis (Archbishop) on Oct 17, 2013 at 01:35 UTC
    Could it be that it's too large for perl? or it could be a non-numeric variable in the data file, so how would I test for that?

    You can get a result of "NaN" by doing addition *only* if one (or more) of the values being summed is a NaN.
    I would therefore check the terms being added to see if any of them match the regex /^nan/i .

    Cheers,
    Rob
[reply]
      or, if the OP happens to use Math::BigFloat:
      $ perl -MMath::BigFloat -wE 'say Math::BigFloat->new(5) + "foo"'
NaN

      [download]
[reply]
[d/l]
        or, if the OP happens to use Math::BigFloat

        It's interesting that Math::BigFloat decides it has to numify a non-numeric string to NaN ... when everything else numifies such a string to zero.

        Actually, I'm tempted to s/interesting/stupid/, but maybe there's a precedent somewhere that gives some credence to the Math::BigFloat behaviour.

        Cheers,
        Rob
[reply]

Back to Seekers of Perl Wisdom