in reply to Wanting some clarification / opinions on MCE vs Threads

For the curious, here is a version of TGrep.pm by BrowserUk modified to use MCE. Simply remove or comment out "use threads" if Perl lacks support for threads. Threads is not necessary for MCE::Queue.

package TGrep; use strict; use threads; use MCE; use MCE::Queue; our $WORKERS = 4; sub tgrep(&@) { my $workers = $WORKERS; my $code = shift; my @results; my $Qin = MCE::Queue->new( fast => 1 ); my $Qout = MCE::Queue->new( queue => \@results ); my $mce = MCE->new( max_workers => $workers, user_func => sub { $Qout->enqueue( $code->() ? $_ : () ) while local $_ = $Qin- +>dequeue; } )->spawn; $Qin->enqueue( map{ ref $_[0] ? @{ $_ } : $_ } @_ ); $Qin->enqueue( (undef) x $workers ); $mce->run; return wantarray ? @results : \@results; } sub import { no strict 'refs'; my $pkg = caller; *{ $pkg . '::' . $_ } = *{ $_ } for qw[ tgrep ]; } 1;

Thus, new results emerges.

# $N = 1e6 (4 workers) TGrep....: Took 30.264018774 seconds TGrep MCE: Took 15.292614937 seconds (fast => 0) TGrep MCE: Took 7.824651003 seconds (fast => 1)

I was curious about how MCE::Queue compares to Thread::Queue for the demonstration. The fast option is beneficial for an already populated queue.

Replies are listed 'Best First'.
Re^2: Wanting some clarification / opinions on MCE vs Threads
by BrowserUk (Patriarch) on Feb 12, 2015 at 08:52 UTC
    I was curious about how MCE::Queue compares to Thread::Queue for the demonstration.

    Thread::Queue, and indeed anything involving threads::shared is very slow. This is a side effect of the latter's peculiar implementation.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
    In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked