Plus, the line @file = <FILE>; puts the entire file into RAM. A 22MB file translates to, probably, 100MB - 200MB in RAM. Add the overhead of Perl and anything else you're doing and you could be using 300MB+ of RAM. If you're doing this on a smaller machine, you could be swapping memory like crazy, which will kill your program's runtime. (And, it might even kill the machine.)

A much better idea would be to pre-process the file, maybe by loading it into a database. Then, do your work against that. Another idea would be to sort the file. That way, you can short-circuit your processing.

The important thing is that you have a program that works. Optimizing for speed once you have accuracy down is good. Optimizing for accuracy once you have speed down is ... harder.

This is a good example of what people mean when they say, "That solution doesn't scale well.

The problem is that the algorithm you've devised to partition the data by range runs in something like O(n^3) time (or worse), which means that as your dataset grows by n elements, the work being done grows by at least n^3 (again, it might be worse in final analysis). You have a construct of four 'for' loops nested, and then two 'for' loops nested later.

As the dataset grows, you go from a relatively small number of iterations, to billions of iterations, rather quickly, and the program appears to grind to a halt -- in reality it's working really really hard.

Couldn't you predeclare your ranges, and then in a single loop test which ranges each line of data fits into?

Your next problem is that you're slurping the entire file. That means that you must hold the entire 22mb file in memory. Plus, the array you're generating must fit in memory too. You're probably getting to a point where your operating system cries uncle and starts using hard-disk swap space. With a lot of swapfile churning, you also get dreadfully slow performance. It would work a lot better if you could read your data in line by line and process it, one line at a time.

use strict;
use POSIX qw(floor);

my $input = $ARGV[0];
open(FILE, $input) || die "ERROR: Unable to open $input FILE for readi
+ng: $!\n";

my @range2;

my ($phi, $psi);
while (<FILE>) {
   $phi = substr($_, 0, 6);
   $psi = substr($_, 8, 6);

   print "$phi $psi\n";
   my $i = floor($phi/5.0) + 36;
   my $j = floor($psi/5.0) + 36;
   if ($i >= 0 && $j >= 0 && $i < 73 && $j < 73) {
      ++$range2[$i][$j];
   }
}

for my $i (0..72) {
   for my $j (0..72) {
      my $val = $range2[$i][$j];
      $val ||= 0;
      print "$val, ";
   }
   print "\n";
}
[download]

I agree -- this sort of approach makes a lot more sense, and will scale very well to any amount of input. I'd just make one minor adjustment:

while (<FILE>) {
#  $phi = substr($_, 0, 6);
#  $psi = substr($_, 8, 6);

   ( $phi, $psi ) = split;
   ...
[download]

The OP didn't show us any lines of data with single-digit numbers in one or both columns. Such lines might or might not have fixed-width fields, and if not, substr() would do the wrong thing; split() would always do the right thing.

(For that matter, it's not clear whether the column separator is two spaces or a single tab character -- again, split works in any case, while substr would get tripped up when you guess wrong about that.)

Firstly, you slurp the entire file into memory with the line that reads @file = <FILE>;. Secondly, you loop over each line of the file many many times. Also, not relating to your problem per se, but you are using an array that contains a single array reference and it doesn't look like there's any reason for that (@ranges will always have one element, the reference to your 4 values. Similar for @range_counts since it's built from @ranges).

@file = <FILE>;
[download]