mgdude has asked for the wisdom of the Perl Monks concerning the following question:

I'm in the process of moving a bunch of Perl scripts from an old WinNT server to a new Unix server. Most of the moving has gone without any problems, Perl is pretty cross-platform compatible. However, certain scripts are giving me trouble.

For example, one script reads numbers from a "|" delimited flat file, then adds up some of the numbers. The data is typically read in as strings and is forced into a number context. Problem is that some strings that apparently have some kind of a leading space character " " in front of the number (ex. " 7") are read as having a numerical value of 0. So when I add this string to a $total, the $total does not change. This strangely occurs only on the Unix box and with only some numbers that have leading spaces, not all. I found a way around the problem - substitute out all the space characters before adding. But, I'm still curious if anyone knows what's going on here... just in case this problem surfaces again in some other place on some other script. Here's a lil' bit o' code:

  

# ex. one database file - 
if (-e $database) {
  open(DB, $database) or die "Error opening file: $!\n";
  while(<DB>){
    push @results, $_;
  }
  close (DB);
  foreach $entry (@results){

    # at this pt. $p_data[3] may look like " 7"
    my ($key, @p_data) = split(/\|/,$entry);

    # $p_data[3] sometimes added to $total w/ numerical value of 0
    $total += $p_data[3];
  }
  # do some other stuff with $total... blah, blah, blah
}

[download]
Thanks, Carl

Replies are listed 'Best First'.
Re: Numerical value of strings
by cianoz (Friar) on Aug 05, 2003 at 15:15 UTC
    maybe there is something more than a space in your string (if you use -w you should have a warning "Argument ? isn't numeric in addition") however you can always do 
    $p_data[3] =~ s/\D//g;
$total += $p_data[3];

    [download]
[reply]
[d/l]
Re: Numerical value of strings
by broquaint (Abbot) on Aug 05, 2003 at 15:13 UTC
    Instead of avoiding the whitespace, just grab the number. Also, are you sure you need to read in the contents of the file, as simple iterating over the file will probably be fine e.g 
    open(DB, $database) or die "Error opening file: $!\n";

my $total = 0;
while(<DB>) {
  my($val) = (split /\|/, $entry)[3] =~ /(\d+)/;
  $total += $val;
}

close DB;

    [download]
    That should iterate over your $database file, split on pipe characters, match for digits on the 4th column and then add that to $total.
    HTH

    _________
    broquaint

[reply]
[d/l]
      You might want to change your regex to  /(-?\d+(?:\.\d*)?)/ in case there is a decimal or negative number in there and not just all positive integers.

      --

      flounder

[reply]
[d/l]
        flounder99,
        While this increases the possibilities there are still some missing. If I were tackling this problem, I would do one of the following:
      • Correct whatever mechanism was causing/allowing the leading spaces "Malformed UTF-8 character"
      • Write a util for stripping the leading spaces "Malformed UTF-8 characters" and saving changes before running the secondary script
      • Delegate the number determination to a modified form of the looks_like_a_number from Scalar::Util. See copy/pasted excerpt below: 
        sub looks_like_number {
  local $_ = shift;

  # checks from perlfaq4
  return 1 unless defined;
  return 1 if (/^[+-]?\d+$/); # is a +/- integer
  return 1 if (/^([+-]?)(?=\d|\.\d)\d*(\.\d*)?([Ee]([+-]?\d+))?$/); # 
+a C float
  return 1 if ($] >= 5.008 and /^(Inf(inity)?|NaN)$/i) or ($] >= 5.006
+001 and /^Inf$/i);

  0;
}

        [download]
        In all honesty, I would do steps 1 and 2 once and incorporate 3 as part of my regular script to ensure my data didn't get polluted with erroneous entries.

        Cheers - L~R

[reply]
[d/l]
Re: Numerical value of strings
by RMGir (Prior) on Aug 05, 2003 at 15:18 UTC
    Edition 3 of Programming Perl says on page 59 that "To convert from string to number, Perl internally uses something like the C library's atof(3) function."

    It's possible that the perl on the unix box giving you problems is using a buggy (or stricter) atof.

    I think the atof implementation is now rolled into perl, but I'm not sure. On older versions, it was probably a library call.

    Edit: HAHA! Reading the p5p summary on use.perl.org today, I got this Larry Wall quote as a random tagline:

    There are many times when you want it to ignore the rest of the string just like atof() does. Oddly enough, Perl calls atof(). How convenient. :-)
    --Larry Wall in <1991Jun24.231628.14446@jpl-devvax.jpl.nasa.gov>

    It's a very old quote, so perl might not use atof any more, but it was still a funny (to me) coincidence.
    --
    Mike

[reply]
Re: Numerical value of strings
by mgdude (Novice) on Aug 05, 2003 at 15:33 UTC
    Thanks, I checked the error log and I am recieving an error: "Malformed UTF-8 character (unexpected continuation byte 0xa0, with no preceding start byte) in addition (+) at blah.pl line 224." Anyone have an idea how that got there?
[reply]
      I'd guess perl 5.8.0 on a Redhat Linux box.

      By default, Redhat sets the locale environment variables to something like 

      LANG="en_US.UTF-8"

      [download]
      Unfortunately, that makes 5.8.0 treat all streams as unicode, which is a bad choice (and corrected in the 5.8.1 release candidates).

      Change your profile to set LANG=en_US, and the problem should go away.

      If I have the OS and perl version right, that is...
      --
      Mike

[reply]
[d/l]
Re: Numerical value of strings
by thinker (Parson) on Aug 05, 2003 at 15:25 UTC

    Hi mgdude

    I can't replicate this on my RedHat 9 machine. 

    my total = 0;
my @arr = (" 7", 8, " 9", 10, 11 , " 12 ");
$total += $_ for @arr;
print "$t\n";

    [download]

    outputs 57, as expected

    cheers

    thinker

[reply]
[d/l]

Back to Seekers of Perl Wisdom