in reply to Re^4: Multi-threads newbie questions
in thread Multi-threads newbie questions
Why doesn't this work?
Really? Works for me:
#! perl -slw use strict; use threads; use threads::shared; use Thread::Queue; use Data::Dump qw[ pp ]; sub helper { my $Q = shift; while( my $ref = $Q->dequeue ) {; lock $ref; $ref->{NEW_KEY} = 1; } } sub my_sub { my( $ref, $n ) = @_; my $Q = new Thread::Queue; my @threads = map async( \&helper, $Q ), 1 .. $n; $Q->enqueue( values %{ $ref } ); $Q->enqueue( (undef) x $n ); $_->join for @threads; } my $hoh = { A => { NAME => 'aa' }, B => { NAME => 'bb' }, }; $hoh = shared_clone( $hoh ); pp $hoh; my_sub( $hoh, 2 ); pp $hoh; __END__ C:\test>junk39 { A => { # tied threads::shared::tie NAME => "aa", }, B => { # tied threads::shared::tie NAME => "bb", }, } { A => { # tied threads::shared::tie NAME => "aa", NEW_KEY => 1, }, B => { # tied threads::shared::tie NAME => "bb", NEW_KEY => 1, }, }
Second, if I understand correctly, you start a thread for each element of hoh. There are usually a few hundreds elements there, so I guess it's not as good idea. that's why I originally wanted you use a thread pool, until you advised me otherwise.
I didn't advice you against using a pool of threads. Only against modules that purport to make using a thread pool "simple", in the most complicated (and broken) ways imaginable.
The code above (also posted at 860833) implements a pool of threads. I posted the non-pooled version first, in order to show how simple the transition from a non-pooled to a pooled solution is. How one is a very small extension of the other.
And why I've never written or used a module to do it. It isn't necessary.
Indeed, the philosophy behind most of those modules is utterly wrong. They manage the number of threads according to how much work is in the queue, for each thread!?! Which is a nonsense, because the number of cores doesn't vary. The processing power of the CPU doesn't vary.
So, at exactly the moment when the CPU is already overloaded--as indicated by the fact the the queue is growing faster than the existing pool of threads can keep up with, what do they do? They start another thread!
Which just means that they've stolen a bunch of cycles to start that thread. And now there is one more thread competing for resources, and having to be task switched by the system scheduler. Which simply slows the throughput of all of the threads.
This asinine methodology is aped from "fork pool" modules, which are equally broken for the same reasons.
The above pooling mechanism starts N threads. You the programmer decide how many threads to run by trial and error. For CPU-bound processing, start with N=cores. For IO-bound, try N=cores * 2 or 3 or 4. You'll quickly find a number that works for your application, then make it the default. Move to a different system with more or less cores, adjust it accordingly.
|
|---|
| Replies are listed 'Best First'. | |
|---|---|
|
Re^6: Multi-threads newbie questions
by daverave (Scribe) on Sep 20, 2010 at 14:54 UTC | |
by BrowserUk (Patriarch) on Sep 20, 2010 at 15:58 UTC | |
by daverave (Scribe) on Sep 20, 2010 at 16:44 UTC | |
by BrowserUk (Patriarch) on Sep 20, 2010 at 16:49 UTC | |
by daverave (Scribe) on Sep 20, 2010 at 18:53 UTC | |
|