Update: added pattern matching to the demonstration
If parallel reads is desired, the following demonstration does the same thing. Basically, specifying chunk_size => 1 is all one needs to do for getting MCE to run like Parallel::ForkManager.
#!/usr/local/bin/perl
use strict;
use warnings;
use MCE::Loop;
my $dir = 'logs/*.log.gz';
my @files = sort(glob "$dir");
my $pattern = "some_string";
MCE::Loop::init {
max_workers => 24,
chunk_size => 1,
};
mce_loop {
my ( $mce, $chunk_ref, $chunk_id ) = @_;
open( my $fh, "-|", "zcat", $chunk_ref->[0] ) or die "open error:
+$!\n";
while ( my $line = <$fh> ) {
if ( $line =~ /$pattern/ ) {
my @matches = $line =~ /".*?"|\S+/g;
print "$matches[0],$matches[1],$matches[3],$matches[4]\n";
}
}
close $fh;
} @files;
Regards, Mario