Greetings, fellow Monks.

I came across an old thread. One might do the following to consume extra CPU cores. The pigz binary is useful and depending on the data, may run faster than gzip. The requirement may be to have each MCE worker process a single file inside the MCE loop. So we set chunk size accordingly (chunk_size => 1).

To make this more interesting, workers send data to STDOUT and gather key-value pairs.

use strict;
use warnings;
use feature qw(say);

use MCE::Loop chunk_size => 1, max_workers => 4;

my @files  = glob '*.gz';

my %result = mce_loop {
    my ($mce, $chunk_ref, $chunk_id) = @_;

    ## $file = $_; same thing when chunk_size => 1

    my $file = $chunk_ref->[0];

    ## http://www.zlib.net/pigz/
    ## For pigz, we want -p1 to run on one core only.
    ## open my $fh, '-|', 'pigz', '-dc', '-p1', $file or do { ... }

    open my $fh, '-|', 'gzip', '-dc', $file or do {
        warn "open error ($file): $!\n";
        MCE->next();
    };

    my $count = 0;

    while ( my $line = <$fh> ) {
        $count++;  # simulate filtering or processing
    }

    close $fh;

    ## Send output to the manager process.
    ## Ensures workers do not garble STDOUT.

    MCE->say("$file: $count lines");

    ## Gather key-value pair.

    MCE->gather($file, $count);

} @files;

## Workers may persist after running. Request workers to exit.

MCE::Loop->finish();

## Ditto, same output using gathered data.

for my $file (@files) {
    say "$file: ", $result{$file}, " lines";
}

[download]

Regards, Mario.

Replies are listed 'Best First'.
Re: Fast gzip log reader with MCE
by marioroy (Prior) on Mar 02, 2017 at 08:56 UTC

    Greetings,

    The demonstration posted above runs on Perl 5.8. The following example requires Perl 5.10.1, minimally. The difference is that this one constructs a shared variable for workers to store results. In regards to MCE::Shared, the OO interface is preferred for maximum performance.

    use strict;
use warnings;
use feature qw(say);

use MCE::Loop chunk_size => 1, max_workers => 4;
use MCE::Shared;

my @files  = glob '*.gz';
my $result = MCE::Shared->hash();

mce_loop {
    my ($mce, $chunk_ref, $chunk_id) = @_;

    ## $file = $_; same thing when chunk_size => 1

    my $file = $chunk_ref->[0];

    ## http://www.zlib.net/pigz/
    ## For pigz, we want -p1 to run on one core only.
    ## open my $fh, '-|', 'pigz', '-dc', '-p1', $file or do { ... }

    open my $fh, '-|', 'gzip', '-dc', $file or do {
        warn "open error ($file): $!\n";
        MCE->next();
    };

    my $count = 0;

    while ( my $line = <$fh> ) {
        $count++;  # simulate filtering or processing
    }

    close $fh;

    ## Send output to the manager process.
    ## Ensures workers do not garble STDOUT.

    MCE->say("$file: $count lines");

    ## Store key-value pair.

    $result->set($file, $count);

} @files;

## Workers may persist after running. Request workers to exit.

MCE::Loop->finish();

## Ditto, same output using shared data.

for my $file (@files) {
    say "$file: ", $result->get($file), " lines";
}

    [download]

    Regards, Mario.

[reply]
[d/l]

Back to Cool Uses for Perl