Below, an example using MCE::Flow. It chunks and outputs swiftly. The relay option when defined loads MCE::Relay and with that enables relay capabilities. Relay is beneficial in places where workers must run orderly and serially. Only a single worker can run inside the relay block. The important thing is that workers enter it orderly by "chunk id" value.

If threads is desired on a Unix platform, simply load threads prior to loading MCE. By default, MCE spawns threads if present. Unlike MCE.pm where chunk_size defaults to 1, chunk_size is configured automatically for MCE models.

use strict;
use warnings;
use MCE::Flow;

## Make gzip file

{
    open my $fh, '|-', 'gzip > test.txt.gz';

    foreach (1..100000) {
        print {$fh} sprintf('%04d',$_).('abc123' x 10)."\n";
    }

    close $fh;
}

## Read gzip file

open my $fh, '-|', 'gzip -cd test.txt.gz' or die "open error: $!\n";

STDOUT->autoflush(1);  # important

MCE::Flow->init(
    max_workers => 3,
    init_relay  => 0
);

sub test {
    my ($mce, $chunkref, $chunkid) = @_;
    my ($buf, $wid) = ('', MCE->wid());

    for my $i (0 .. $#{ $chunkref }) {
        $buf .= $chunkref->[$i];
    }

    MCE::relay {
        print "## worker: $wid, chunkid: $chunkid\n".$buf;
    };
}

MCE::Flow->run_file(\&test, $fh);
MCE::Flow->finish();

close($fh);

[download]

Cheers, Mario.

In reply to Re^7: shared scalar freed early by marioroy
in thread shared scalar freed early by chris212

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.