in reply to An efficient, scalable matrix transformation algorithm

Luftkissenboot:

If you can make all your consolidation functions able to work incrementally, you could process your data one row at a time. Even if you can't make the function work incrementally, you might be able to transform it such that you can use partial results and combine them at the end (last line of example before __DATA__ statement). Something like this:

#!/usr/bin/perl -w # # Sploink.pl # use strict; use warnings; sub make_summer { my $sum = 0; return sub { $sum += $_[0]; } } sub make_minner { my $min = 9.99e99; return sub { $min = $_[0] if $_[0] < $min; return $min; } } sub make_avger { my ($cnt, $sum) = (0, 0); return sub { $sum += $_[0]; ++$cnt; return $sum/$cnt; } } sub make_counter { my $cnt = 0; return sub { ++$cnt } } my @funcs = ( make_summer(), make_minner(), make_avger(), make_avger(), make_counter(), ); my @results; while (<DATA>) { chomp; my @flds = split /\s+/; $results[$_] = $funcs[$_]($flds[$_]) for (0 .. $#funcs); } print "results: ", join(", ", @results), "\n"; print "avg of column 1 is: ", $results[0] / $results[4], "\n"; __DATA__ 10 20 30 40 12 14 16 18 9 10 11 12 13 14 15 16

Which gives the following:

roboticus@swill: /Work/Perl/PerlMonks $ ./sploink_739466.pl results: 44, 10, 18, 21.5, 4 avg of column 1 is: 11

...roboticus

UPDATE: Added program output

Replies are listed 'Best First'.
Re^2: An efficient, scalable matrix transformation algorithm
by Luftkissenboot (Novice) on Jan 29, 2009 at 07:30 UTC

    Actually, that is very feasible. Now that I've dug further into it from previous conversations also, it would lend itself to it, and it would be a major memory saver, to boot.

    So many good suggestions. Thank you!