in reply to Multi-thread combining the results together

Something like this:

use strict;
use warnings;
use threads;
use Thread::Queue;

my $workQueue = Thread::Queue->new();
my $doneQueue = Thread::Queue->new();
my @threads;

push @threads, threads->create(sub{DoWork($workQueue, $doneQueue)}) fo
+r 1 .. 4;

for (1 .. 10000) {
    my $work = int(rand(10000));

    $workQueue->enqueue($work);
}

$workQueue->end();
$_->join() for @threads;

print "$_\n" while $doneQueue->pending() && ($_ = $doneQueue->dequeue(
+));
exit;

sub DoWork {
    my ($work, $done) = @_;

    while (my $item = $work->dequeue()) {
        $done->enqueue($item) if $item >= 10 && $item <= 20;
    }
}

[download]

Prints (YMMV):

17
14
11
19
12
10
13
10
11

[download]
Optimising for fewest key strokes only makes sense transmitting to Pluto or beyond

Replies are listed 'Best First'.
Re^2: Multi-thread combining the results together
by marioroy (Prior) on Jul 25, 2019 at 11:58 UTC

    If your Perl is not built with threads support, no problem. GrandFather's queue demonstration is possible using MCE::Hobo and MCE::Shared. Notice the similarity.

    use strict;
use warnings;
use MCE::Hobo;
use MCE::Shared;

my $workQueue  = MCE::Shared->queue();
my $doneQueue  = MCE::Shared->queue();
my $numWorkers = 4;

MCE::Hobo->create(sub{DoWork($workQueue, $doneQueue)}) for 1 .. $numWo
+rkers;

for (1 .. 10000) {
    my $work = int(rand(10000));
    $workQueue->enqueue($work);
}

$workQueue->end();

my $count_done = 0;

while () {
    my $result = $doneQueue->dequeue();
    last if (!$result && ++$count_done == $numWorkers);
    print "$result\n" if $result;
}

$_->join() for MCE::Hobo->list;

exit;

sub DoWork {
    my ($work, $done) = @_;
    while (my $item = $work->dequeue()) {
        $done->enqueue($item) if $item >= 10 && $item <= 20;
    }
    $done->enqueue(0);
}

    [download]

    Regards, Mario

[reply]
[d/l]
Re^2: Multi-thread combining the results together
by Marshall (Canon) on Jul 31, 2019 at 03:48 UTC
    Thank you very much!

    I did have to merge some of your ideas together with TreadQueue.
    My code works and is 3+x faster with 4 threads. That is about what I expected in terms of performance increase.

    Thanks again - your code is much clearer than the Perl Doc and is a much easier to follow set of model code.

    Update: some code:

    Read more... (4 kB)
[reply]
[d/l]
Re^2: Multi-thread combining the results together
by Marshall (Canon) on Aug 02, 2019 at 05:10 UTC
    Thank you Grandfather! This was perfect!

    This was a lot easier for me to understand than one example I saw in the docs. I was able to successfully adapt my code and meet my anticipated goal of 30 min with 4 cores. That is fast enough for me to continue on with the analysis work that I'm doing right now. I'd like to develop a better algorithm, but that will be the subject of another SOPW question - There has got to be some kind of tree structure that will work much, much faster but I'm not sure how to build something like that with wildcards. Thanks Again!

    Oops, I see that I already posted a thank you. So this is redundant.

[reply]
Re^2: Multi-thread combining the results together
by Anonymous Monk on Jul 25, 2019 at 07:46 UTC
    Hi join is blocking
[reply]

      Anonymous Monk:

      Yes, but that's done after all the work has been queued. So the main thread simply waits for all the work to complete before exiting.

      In a "real" application, that would let your main thread collect some statistics and report before ending, to let you know what's going on.

      ...roboticus

      When your only tool is a hammer, all problems look like your thumb.

[reply]

      In this similar example, joining (reaping workers) is done after obtaining results.

[reply]
Re^2: Multi-thread combining the results together
by Anonymous Monk on Aug 02, 2019 at 07:43 UTC
    No need for extra wrappers 
    push @threads, threads->create(\&DoWork, $workQueue, $doneQueue) for 1
+ .. 4;

    [download]
[reply]
[d/l]

In Section Seekers of Perl Wisdom