Re^5: Threads From Hell #1: How To Share A Hash [SOLVED]

Your diagram (whilst probably correct for this case) is too prescriptive. It implies that the work is dolled out like kids with Smarties: 1 for you, 1 for you...

But imagine (or try) that the list of numbers was 1000 randomly chosen from 1 .. 2000, and in any order.

Then you'd see that one thread might successively happen to get runs of small (say 1 or 2 digit) numbers that take little time; whilst another thread might happen to always get 4-digit numbers that take longer; with the result that by the end, the first thread may have processed many more items than the second.

But they will both have processed nearly the same number of clock cycles and will end within a few milliseconds or so of each other.

That's one of the beauties of the queue architecture is that it is inherently self-balancing:

#! perl -slw
use strict;
use threads;
use threads::shared;
use Thread::Queue;
use Math::BigInt;
use List::Util qw[ shuffle ];
use Time::HiRes qw[ time ];
use Data::Dump qw[ pp ];
use feature qw(say);

our $T //= 4;

my %result : shared;

sub process {
    my $Q = shift;
    my $n = 0;
    while( my $number = $Q->dequeue ) {
        my $factorial = factorial($number);
        lock %result;
        $result{$number} = $factorial->bstr;
        ++$n;
    }
    printf "[%2u] terminating at %f; having processed %u values\n", th
+reads->tid, time(), $n;
}

sub factorial {
    my $number = shift;
    Math::BigInt->bfac($number);
}

my $start = time;

my $Q = new Thread::Queue;
$Q->enqueue( (shuffle 1..2000 )[ 0 .. 999 ], (undef) x $T );

my @threads = map { threads->create( \&process, $Q ); } 1 .. $T;

$_->join for @threads;
my $end = time;
printf "Took %.6f seconds\n", $end - $start;
[download]

Outputs:

C:\test>1126584-q
[ 4] terminating at 1431869328.993618; having processed 280 values
[ 1] terminating at 1431869329.076621; having processed 305 values
[ 3] terminating at 1431869329.094198; having processed 190 values
[ 2] terminating at 1431869329.102986; having processed 225 values
Took 17.553564 seconds
[download]

The only guarantee is that eventually all the work items are processed and each of the 4 threads will dequeue an undef and so end its loop.

It does mean that work items cannot themselves be undef (or any non-true value unless you modify the loop to: while( defined( my $number = $Q->dequeue ) ){).

And later versions of Thread::Queue acquired, (amongst a lot of dross), an end() method which can be used as an alternative to queueing undef's, but the implementation leaves a lot to be desired; and old (tried and tested) programming habits die hard :)

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority". I'm with torvalds on this

In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked

Comment on Re^5: Threads From Hell #1: How To Share A Hash [SOLVED] Select or Download Code

Replies are listed 'Best First'.
Re^6: Threads From Hell #1: How To Share A Hash [SOLVED] by karlgoethebier (Abbot) on May 17, 2015 at 14:29 UTC
"But imagine (or try)..." I'll make it so. But for the moment i feel a bit like i re-opened Pandora's box. One thing comes into my mind: "Few people know how to use them correctly or will be able to provide help" Thank you very much and my best regards, Karl ŤThe Crux of the Biscuit is the Apostropheť	[reply]
Re^7: Threads From Hell #1: How To Share A Hash [SOLVED] by BrowserUk (Patriarch) on May 17, 2015 at 15:07 UTC
i feel a bit like i re-opened Pandora's box. If you read Friedl's book before ever using regex; you might never try, because the subject seems so complicated, but millions of people get by without major problems, having never read it. If your program requires you use a lot of complex regex, and performance is a high criteria, then reading the book will save you heaps of time, experiments and blind alleys, but it's not a good starting point. And it is a similar thing with threading; there is so much folk-law out there saying they are complicated; many people are put off from even trying; but if you start with simple requirements and progress naturally, they aren't hard to make effective use of; even if extracting the last ounce of performance from them is something of a trial by fire to get right. It's not helped by the continued existence of so many of the early, failed attempts, to 'simplify' them into 'patterns'. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked .	[reply]
Re^8: Threads From Hell #1: How To Share A Hash [SOLVED] by karlgoethebier (Abbot) on May 17, 2015 at 17:23 UTC
"If you read Friedl's book before ever using regex; you might never try..." Fortunately this splendid book wasn't at my disposal when i started using Perl. But if so i would have thrown it out of the window. In the next step i would have deleted everything related to Perl from my box. Thanks for motivation to continue and best regards, Karl ŤThe Crux of the Biscuit is the Apostropheť	[reply]