Hi Pascal666,

The following are three parallel demonstrations using MCE::Flow, MCE::Hobo, and threads. The shared input and output handles are managed by MCE::Shared.

MCE::Flow and MCE::Shared

Many-Core Engine provides chunking abilities not used here.

use strict;
use warnings;

use MCE::Flow;
use MCE::Shared;
use PerlIO::gzip;

my $IN  = MCE::Shared->handle( '<:gzip', 'wat.paths.gz' );
my $OUT = MCE::Shared->handle( '>', \*STDOUT );

mce_flow { max_workers => 12 }, sub {
   while (my $file = <$IN>) {
      print $OUT length($file).":$file" if length($file) > 142;
   }
};

close $IN;
[download]

MCE::Hobo and MCE::Shared

A Hobo is a migratory worker inside the machine that carries the asynchronous gene. Hobos are equipped with threads-like capability for running code asynchronously. Unlike threads, each hobo is a unique process to the underlying OS. The IPC is managed by MCE::Shared, which runs on all the major platforms including Cygwin.

use strict;
use warnings;

use MCE::Hobo;
use MCE::Shared;
use PerlIO::gzip;

my $IN  = MCE::Shared->handle( '<:gzip', 'wat.paths.gz' );
my $OUT = MCE::Shared->handle( '>', \*STDOUT );

sub task {
   while (my $file = <$IN>) {
      print $OUT length($file).":$file" if length($file) > 142;
   }
}

MCE::Hobo->create('task') for 1 .. 12;

# do other stuff if desired

$_->join for MCE::Hobo->list;

close $IN;
[download]

threads and MCE::Shared

The code for MCE::Hobo and threads are very similar.

use strict;
use warnings;

use threads;
use MCE::Shared;
use PerlIO::gzip;

my $IN  = MCE::Shared->handle( '<:gzip', 'wat.paths.gz' );
my $OUT = MCE::Shared->handle( '>', \*STDOUT );

sub task {
   while (my $file = <$IN>) {
      print $OUT length($file).":$file" if length($file) > 142;
   }
}

threads->create('task') for 1 .. 12;

# do other stuff if desired

$_->join for threads->list;

close $IN;
[download]

All three examples work across the board including with Perl on Windows and Cygwin, provided the OS has the gzip binary and the PerlIO::gzip module installed.

Update:

The upcoming MCE::Shared 1.002 release will support the following construction by allowing the main or worker process to handle the error. I've been wanting for the shared open call to feel like the native open call.

use MCE::Shared 1.002;

mce_open my $IN, "<:gzip", "wat.paths.gz" or die "open error: $!";
mce_open my $OUT, ">", \*STDOUT or die "open error: $!";
[download]

Kind regards, Mario

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.