in reply to Calculating the average on a timeslice of data

It's too bad you don't have it in CSV format - you could just use DBD::CSV to get your average ;-) SELECT SUM(VALUE)/COUNT(*) FROM mytable WHERE DATE = ?

If you're feeling really chipper, you could put this data into a real database, and, again, the average will be trivial. While I'm mostly kidding with this one, it really depends on what else you're doing with those 40,000 lines - it may really be cheaper to put it in a database (even SQLite) and use SQL to get your information than to do it yourself. But that's only true if you have more than one query to make against it.

However, if you want to approach it directly, I'm not sure why it isn't feasible to use if ($date eq '0109'). It seems perfectly feasible to me. 

my ($total, $count);
while (<$fh>)
{
  # omitting any error checking here - you shouldn't omit it, though.
  my ($id, $date, $value) = split ' ';
  if ($date eq $desired_date)
  {
    $total += $value;
    ++$count;
  }
}
my $avg = $total / $count;

[download]
As for your idea to load everything into hashes, that's fine, too. The huge disadvantage is the amount of RAM you'll use. You'll spend a bunch of time populating the hash, too. If you're only querying one thing out of it, that's all wasted time and space. If you're making multiple queries in the same process, then you can see a speed benefit from not having to re-read the file every time. It can be faster than a database, but will likely use more RAM, and will need to re-parse the file every time you load your perl script, whereas a database would have indexes that would speed things up across multiple processes. So, again, it all depends on your usage.

Most likely, the above code that scans through the file with the if is more than sufficient.

Replies are listed 'Best First'.
Re^2: Calculating the average on a timeslice of data
by perlbrother (Initiate) on Jul 06, 2011 at 16:13 UTC
    Thanks for the reply and explanation. The reason I was thinking using an if statement is because there are 120 desired dates that I need to calculate an average for, and I didn't want to include 120 if statements in my code. Also the file being read in may be updated each week with new data and new dates, so ideally I wouldn't have to go back into the code and add new if statements to get the average for the new dates
[reply]

      Why would you have multiple if's? You don't hardcode things that vary - that's what variable means. You need a way to figure out what date(s) you want to gather averages for, which may be "all of them", and then vary on that. For example, if you want the user to specify the date, just make it a command line parameter: avg_for.pl 0109. Then you just set my $desired_date = shift @ARGV;, and the code I posted would suffice.

      If you want to produce an average for ALL dates, you can use a hash to accumulate it, and then loop through the keys for the output: 

      use strict;
use warnings;
use Text::Table;

my %totals;
while (<DATA>)
{
    my ($id, $date, $value) = split ' ';
    $totals{$date}{sum} += $value;
    ++$totals{$date}{count};
}

my $tbl = Text::Table->new('Date', \' | ', 'Average');

for my $date (sort keys %totals)
{
    $tbl->add($date, $totals{$date}{sum} / $totals{$date}{count});
}
print $tbl;

__END__
1   0109           4
1   0209           5
1   0309           8
2   0109           6
2   0209           6

      [download]
      And this produces output like this: 
      Date | Average
0109 | 5      
0209 | 5.5    
0309 | 8

      [download]

[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom