So I have this Perl program reading a very large file, building hashes and keeping line number counters. The Ubuntu 20.04 system (on a 16 GB RAM and 4GB swap) stills kills the program because the hash eats the RAM. My workaround is use MCE to loop through the file chunks using 16 parallel workers.

The program builds the hashes and increments the counter alright but problem is the variables inside mce_loop_f{} are not available outside it so the rest of the code can not use the counter to run through the hash and print the hash vals to an output table. How can I get these variables and hashes be available?

I could not find in the many MCE docs a solution to achieve this except a mention of how Perl may re-spawn

It is possible that Perl may create a new code ref on subsequent runs causing MCE models to re-spawn. One solution to this is to declare global variables, referenced by workers, with "our" instead of "my".

use strict;
use warnings;
use MCE::Loop;
use Data::Dumper;

my %hash;
our %hash2;
our $counter1=0;
our $counter2=0;

open (my $fh, "<", "DATA_F") or die($!);
print "printing counter1\n";
while(<$fh>){
    my ($k, $v)=split;
    $hash{$k}=$v;
    print $counter1++,$/;
}
print "done printing counter1\n";

print "printing counter2\n";
MCE::Loop::init {
   use_slurpio => 1, 
   max_workers => 16, init_relay=>0
};

mce_loop_f{
    my ($mce, $file, $id)=@_;
    open (my $ifh, "<", $file) or die("$!");
    while(<$ifh>){
        my ($k, $v)=split;
        $hash2{$k}=$v;
        print $counter2++,$/;
        }

    } $fh;
print "done printing counter2\n";    

print $counter1, "=counter1 final\n";
print $counter2,"=counter2 final\n";

print Dumper(\%hash);
print Dumper(\%hash2);
[download]

printing counter1
0
1
2
done printing counter1
printing counter2
0
1
2
done printing counter2
3=counter1 final
0=counter2 final #How can I get MCE to make $counter2 available to pri
+nt()?
$VAR1 = {
          '2' => 'two',
          '1' => 'one',
          '3' => 'three'
        };
$VAR1 = {};  #How can I get MCE to make %hash2 available to Dumper()?
[download]