in reply to Re: Wanting some clarification / opinions on MCE vs Threads
in thread Wanting some clarification / opinions on MCE vs Threads

I haven't seen any direct comparisons between the two but I think MCE::Grep and MCE::Map are two examples of MCE code that is more readable than the threads equivalent (I should say my threads equivalent)

use MCE::Grep; my @a = mce_grep { $_ % 5 == 0 } 1..10000;
versus
use threads; use Thread::Queue; my @thread_pool; my $q = Thread::Queue->new(); my $results = Thread::Queue->new(); for (0..10000) { $q->enqueue($_); } for (0..1) { push @thread_pool, threads->create( \&grep ); } sub grep { while (my $work = $q->dequeue() ) { if ( $work % 5 == 0 ) { $results->enqueue($work); } } $q->enqueue(undef); } map {$_->join(); } (@thread_pool); $results->enqueue(undef); my @results; while ( my $result = $results->dequeue() ) { print $result, "\n"; push @results, $result; }

I'm sure that the threads version could be done much more easily than I hacked together in 5 minutes. It could just be that how I write using threads is just poor. Regardless, I don't think there is a threads implementation as simple as the MCE version. Additionally, this is also a special case where MCE has a built-in function that provides this functionality, but there are similar constructs for most of the simple cases. For what I do there isn't much that can't be implemented using some mixture of MCE::Grep, MCE::Map and MCE::Loop so I'm biased.

I should also note that I haven't written very much (no "production" code as it were) using MCE so I may not have encountered some of its limitations compared to threads.

Replies are listed 'Best First'.
Re^3: Wanting some clarification / opinions on MCE vs Threads
by BrowserUk (Patriarch) on Feb 05, 2015 at 10:44 UTC
    (I should say my threads equivalent)

    Hm. No offence but, Ew! :)

    Remember that MCE is a wrapper (actually a suite of wrappers, or is that sweet wrapper:) over the top of threads(and other things), providing syntactic sugar for simple operations.

    You can easily do the same yourself. Say write TGrep.pm:

    Then write:

    #! perl -slw use strict; use TGrep; use Time::HiRes qw[ time ]; our $N //= 1e3; my $start = time; my @a = tgrep{ $_ % 5 == 0 } [ 1..$N ]; printf "Took %.9f seconds\n", time() -$start; print scalar @a; __END__ C:\test>t-tgrep -N=1e5 Took 3.830074787 seconds 20000

    Of course, you'd probably be better sticking to grep for such simple things:

    $t=time; my @a = grep{ $_%5 == 0 } 1 .. 1e6; print time-$t;; 0.18144702911377

    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
    In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked
    ? @{ $_ } : $_ } @_ ); $Qin-

      Yeah, I would never use threads for something this simple, I was just demonstrating what I perceived to be a readability improvement between threads and MCE with that example since it was quick to work up.

      Since it is just a wrapper for threads, does it make sense to use it in when possible? Is there any reason not to use it IF I am in a situation where I can install modules? (which I should be since 99% of the time I'm running code on my lab machines). I'm starting to think that there isn't a downside except that MCE is not part of a standard Perl install but there aren't many code examples of MCE in the wild so it's hard to tell.

        Since it is just a wrapper for threads,

        It seems to do more than just wrap threads. Indeed, it appears to spawn processes by default and only uses threads if you ask for them.

        does it make sense to use it in when possible?

        If you're (more) comfortable using this kind of (IMO over-engineered) framework, then take a look at the bug history (short & up to date) and maintenance history (3 new versions in 3 months) and then try it for your application.


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
        In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked
        nm, I understand now
      use Threads::Queue;

      I just wondered why i failed to install Threads::Queue ;-)

      Perhaps i should have better read the code: my( $Qin, $Qout ) = map Thread::Queue->new, 1..2;?

      My best regards, Karl

      «The Crux of the Biscuit is the Apostrophe»

        I just wondered why i failed to install Threads::Queue ;-)

        Whoops! Sorry about that. (Now corrected; thanks.)


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
        In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked
Re^3: Wanting some clarification / opinions on MCE vs Threads ( threads_grep )
by Anonymous Monk on Feb 05, 2015 at 10:21 UTC

    Sure, threads could be extended with some convenience functions

    #!/usr/bin/perl -- use strict; use warnings; use threads ; use Thread::Queue; Main( @ARGV ); exit( 0 ); sub threads_grep(&@) { my $cb = shift; my $max = int @_; my $args = ref $_[0] ? shift : { slaves => 4, maxdq => $max * 10 +00 }; my $slaves = $args->{slaves} || 4; my $maxdq = $args->{maxdq} || 1e9; my $qin = Thread::Queue->new( @_, ( undef ) x $slaves ); my $qout = Thread::Queue->new(); my @kids = map { threads->create( sub { ## threads_grep_cb my( $cb, $qin, $qout ) = @_; local $_; while( $_ = $qin->dequeue ) { if( $cb->() ) { $qout->enqueue( $_ ); } } warn 'tids ahoy ', threads->tid; return; }, $cb, $qin, $qout ); } 1 .. $slaves; $_->join for @kids; $qin->end; $qout->end; return $qout->dequeue( $maxdq ); } ## end sub threads_grep(&@) sub Main { #~ my @res = threads_grep { $_ % 5 == 0 } 1..1000; my @res = threads_grep { $_ % 5 == 0 } { slaves => 2 }, 1..1000; print "@res\n"; } ## end sub Main __END__ tids ahoy 1 at - line 26. tids ahoy 2 at - line 26. 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 105 110 11 +5 120 125 130 135 140 145 150 155 160 165 170 175 180 185 190 195 200 205 210 215 220 225 230 235 240 245 250 255 260 26 +5 270 275 280 285 290 295 300 305 310 315 320 325 330 335 340 345 350 355 360 365 370 375 380 385 390 395 400 405 410 41 +5 420 425 430 435 440 445 450 455 460 465 470 475 480 485 490 495 500 505 510 515 520 525 530 535 540 545 550 555 560 56 +5 570 575 580 585 590 595 600 605 610 615 620 625 630 635 640 645 650 655 660 665 670 675 680 685 690 695 700 705 710 71 +5 720 725 730 735 740 745 750 755 760 765 770 775 780 785 790 795 800 805 810 815 820 825 830 835 840 845 850 855 860 86 +5 870 875 880 885 890 895 900 905 910 915 920 925 930 935 940 945 950 955 960 965 970 975 980 985 990 995 1000