You are reading the full file in memory before processing it. Use while to loop instead of foreach.

Also, you are using a hash of hashes to store the counters when a simple hash will do and reduce the memory consumption an order of magnitude:

@ARGV == 1 or die "Usage: ...";
my $inFile = $ARGV[0];

my %freq;

open(FH, "<$inFile");
while (<FH>) {
  chomp;
  if(/(\w+_meas_.*)/) {
    ##($layer, $enc) = split(' ', $1); 
    $layer =  $1; 
    #print "$layer, $enc\n";
    $freq{$layer}++;  
  }
}

foreach $key (keys %freq) {
  print "$freq{$key} $key\n";
}
[download]