What did you try? How long did it take and how much do you need to speed it up? When you profiled it, where was the slowness?

Update: in case it's not clear, the order is:

1. write some code that does the job
2. If it's fast enough you're done
3. If it's too slow, profile it to figure out where a fix is needed
4. write unit tests to make sure your speed fix doesn't break the code!

It's easy to get into a funky "do nothing" state because you can't figure out the "best" way to do something. Don't. Just pick what seems the easiest option to write and go for it. If it does the job, great. If it doesn't, at least you have something to work with and understand the problem better.

A late TMTOWTDI , perhaps not rock-solid:

#!/usr/bin/env perl

use strict;
use warnings;
use List::MoreUtils qw(natatime);
use List::Util qw(reduce);
use IO::All;
use autodie;

my $file     = shift;
my $amount   = 2;
my $io       = io $file;
my $iterator = natatime $amount, @$io;

while ( my @items = $iterator->() ) {
    my @values = map { /^.+\|(.+)$/ } @items;
    my $sum = reduce { $a + $b } @values;
    my $average = $sum / $amount;
    print qq($average\n);
}

__END__
[download]

Update: I took the title of your OP ("...take 'n' at a time...") literally. Or i tried it, at least ;-)

Regards, Karl

Here is one possible way:

use strict;
my $n = 2;
my $cnt;
my $sum;

while(<>) {
    chomp;
    $cnt++;
    my (undef, $v) = split(/\|/);
    $sum += $v;
    print "cnt: $cnt v: $v sum: $sum\n";
    if (($cnt % $n) == 0) {
        print "avg: " . $sum / $n . "\n";;
        $sum = 0 ;
    }
}
[download]

Consider using natatime from List::MoreUtils, check the provided examples.

I was looking for that and didn't find it...I had an idea that I had seen it but missed it in Google somehow. Thanks

Doesn't sound too demanding computationally. Create a routine that reads n number of records from a given file, adding all the values and dividing by n. Then run it for every file you're interested in.

here's what I think I'll use: (the entire file is in @array)

my @i;

while (@array) {
        last if (scalar @array < 9);
        push @i, shift @array for 0 .. 8;
        print join ("\n", @i);
        print "\n------------------------------------------\n";
        @i = ();
        }
[download]

Thanks

Hi,

Well that code is a step in the right direction. Couple things though, you can put the test inside the while loop instead of using "last". You can grab the first 9 items off of the array and delete them from the array in one operation using splice. This lets you avoid using the "@i" array. So:

my @array = map {$_ * 10} 1 .. 20;   # test data
while (@array > 8) {
        print join "\n", splice(@array,0,9);
        print "\n--------------------\n";
}
[download]

Note that if the array isn't divisible by 9, the last few items will be silently ignored. If you'd rather see them, it's safe to just omit the "> 8".. splice will return whatever remains even though it's not the full 9 items.

If 'n' equates to every hour,day,month etc then I would think database.