in reply to Threads From Hell #1: How To Share A Hash [SOLVED]

Corion kindly advised my not to use Thread::Semaphore and pointed me to ... Thread::Queue.

I finally got around to putting together proof of the wisdom of Corion's advice.

This is a version of your Thread::Semaphore code, limiting to 4 concurrent threads and calculating factorials 1000! .. 2000!:

#! perl -slw
use strict;
use threads;
use threads::shared;
use Thread::Semaphore;
use Math::BigInt;
use Time::HiRes qw[ time ];
use Data::Dump qw[ pp ];
use feature qw(say);

my %result : shared;
my $semaphore = Thread::Semaphore->new(4);

sub process {
    my $number    = shift;
    my $factorial = factorial($number);
    lock %result;
    $result{$number} = $factorial->bsstr;
    $semaphore->up;
}

sub factorial {
    my $number = shift;
    Math::BigInt->bfac($number);
}

my @numbers = ( 1000..2000 );

my $start = time;
my @threads = map {
    $semaphore->down;
    threads->create( \&process, $_ );
} @numbers;

$_->join for @threads;
my $end = time;
printf "Took %.6f seconds\n", $end - $start;

[download]

Total runtime: Took 47.511607 seconds

This version does the same calculations using the same number of concurrent threads, but ditches Thread::Semaphore in favour of Thread::Queue to queue the 1000 numbers to 4 reused threads, thus saving the startup and teardown costs of 996 threads:

#! perl -slw
use strict;
use threads;
use threads::shared;
use Thread::Queue;
use Math::BigInt;
use Time::HiRes qw[ time ];
use Data::Dump qw[ pp ];
use feature qw(say);

my %result : shared;

sub process {
    my $Q = shift;
    while( my $number = $Q->dequeue ) {
        my $factorial = factorial($number);
        lock %result;
        $result{$number} = $factorial->bstr;
    }
}

sub factorial {
    my $number = shift;
    Math::BigInt->bfac($number);
}

my $start = time;

my $Q = new Thread::Queue;
$Q->enqueue( 1000..2000, (undef) x 4 );

my @threads = map { threads->create( \&process, $Q ); } 1 .. 4;

$_->join for @threads;
my $end = time;
printf "Took %.6f seconds\n", $end - $start;

[download]

Total runtime: Took 31.290966 seconds; giving a 33% saving of time.

But the biggest lesson of threading, is when not to use it. This version ditches threads altogether and uses the obvious optimisation:

#! perl -slw
use strict;
use Math::BigInt;
use Time::HiRes qw[ time ];
use feature qw(say);

my $start = time;

my $bn = Math::BigInt->bfac( 1000 );
my %result;
$result{ 1000 } = $bn->bstr;

for( 1001..2000 ) {
    $bn *= $_;
    $result{ $_ } = $bn->bstr;
}

my $end = time;
printf "Took %.6f seconds\n", $end - $start;

[download]

Total runtime: Took 0.685503 seconds. Same results, but a 98.5% time saving over the first version above.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked

Replies are listed 'Best First'.
Re^2: Threads From Hell #1: How To Share A Hash [SOLVED]
by karlgoethebier (Abbot) on May 15, 2015 at 18:34 UTC
    "But the biggest lesson...the obvious optimisation..."
    "Du sprichst ein großes Wort gelassen aus." (Johann Wolfgang von Goethe)

    Very impressive and instructional. What else should i say?

    Update:

    There are two little things i would like to add:

    My basic theme wasn't to calculate factorials using threads.

    I wanted to explore multithreading and thought the best idea would be to use some "expensive" calculation for this. Like in a classic homework.

    Unfortunately you provided a much better algorithm that avoids multithreading ;-)

    I fear i hit a basic problem in this context.

    And apropos little quirks:

    I wrote:

    sub process {
    my $number = shift;
    # lock %result;
    $result{ threads->tid() } =
      shared_clone( { $number => factorial($number) } );
    $semaphore->up;
}

    [download]

    It seems like i really need to say something like:

    for my $key ( sort { $a <=> $b } keys %result ) {
    my $ref = $result{$key};
    while ( my ( $k, $v ) = each %{$ref} ) {
        say qq($k => $v);
    }
}

    [download]

    ...to iterate over the results. From the docs:

    "each HASH does not work properly on shared references embedded in shared structures"

    Thanks for this advice and my best regards, Karl

    «The Crux of the Biscuit is the Apostrophe»

[reply]
[d/l]
[select]
Re^2: Threads From Hell #1: How To Share A Hash [SOLVED]
by karlgoethebier (Abbot) on May 16, 2015 at 11:31 UTC

    I'm struggling with this:

    $Q->enqueue( 1000..2000, (undef) x 4 );

    [download]

    The docs say:

    ->enqueue(LIST)
Adds a list of items onto the end of the queue.

    [download]

    I dumped it:

    karls-mac-mini:monks karl$ ./queue.pl 
bless({
  # tied threads::shared::tie
  queue => [
    # tied threads::shared::tie
    1000 .. 2000,
    undef,
    undef,
    undef,
    undef,
  ],
}, "Thread::Queue")
Took 13.820966 seconds

    [download]

    What are the four undefs good for?

    Best regards, Karl

    P.S.: But my box is faster than yours ;-)

    «The Crux of the Biscuit is the Apostrophe»

[reply]
[d/l]
[select]
      What are the four undefs good for?

      They cause these loops to terminate:

      while( my $number = $Q->dequeue ) {

      [download]
      P.S.: But my box is faster than yours ;-)

      No surprise there. My Q6600 is over 7 years old.

      Mind you, I make better use of it than most people do of their Haswells :)

      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority". I'm with torvalds on this
      In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked
[reply]
[d/l]
        "Mind you, I make better use of it than most people do of their Haswells"

        Indeed.

        "They cause these loops to terminate"

        I guess it works like this:

        | 1000  1001  1002  1003 |
| 4 threads ------------ |

| 1004  1005  1006  1007 |
| 4 threads ------------ |

... until they meet undef

| undef undef undef undef|
| 4 threads ------------ |

        [download]

        Best regards, Karl (still locked/blocked)

        «The Crux of the Biscuit is the Apostrophe»

[reply]
[d/l]

In Section Seekers of Perl Wisdom