in reply to Perl Threads Boss/Worker Example

Rather than creating/destroying worker threads for each file, maybe try using a queue? Each worker thread looks for items in the queue, and work on them, avoiding your sleep()s (I hate sleep in my code - it's usually a sign that I did something wrong). You can then join on all the threads at the end, which is simpler.

See the docs for Thread::Queue. You'll end up with something like this:


my $q = Thread::Queue->new();    # A new empty queue
 
# Worker threads
my @thr = map {
  threads->create(sub {
    while (my $item = $q->dequeue()) {
      # assuming undef isn't a valid value and so can be a marker.
      return unless defined $item;
      do_stuff($item, @_);
    }, $param1, $param2
   )->detach();
} 1..$thread_limit;
 
# Send work to the threads
$q->enqueue($_) for @array_of_files;

# send markers.
$q->enqueue(undef) for 1..$thread_limit;

# terminate.
$_->join() for @thr;

[download]
In this case, you can pretty much not count your main thread: after it finishes enqueueing all the files, it will only wake up once per worker thread. So, if you have a quad-core, you really can have a thread limit of 4 instead of 3. That can speed things up a bit as well. Also, you may be able to tweak it a bit - depending on how much each thread spends in I/O vs CPU, you may be able to use 5 or even more threads. Of course, you may throttle your disk at this point, so you may find your disk spinning at full tilt while your CPU usage still doesn't peg at full. At that point, yes, SSD is probably your next best bet for improving the speed.

Replies are listed 'Best First'.
Re^2: Perl Threads Boss/Worker Example
by sunkid (Initiate) on Jul 11, 2013 at 02:47 UTC
    I realize that this is an old thread but I ran across this trying to solve a similar problem: run a specific number of threads to work on a set of issues. I thought it might be useful for others to know the correct implementation of Tanktalus' suggestion.

    Basically, there were two issues that took me a while to sort out: firstly, the newly created threads are detached, which makes them impossible to join later; secondly (much less important), adding undef as a queue item isn't necessary when using dequeue_nb instead of dequeue.

    The code that actually works for me looks like this: 

    #! /usr/bin/perl
use threads;
use Thread::Queue;

my $q = Thread::Queue->new();    # A new empty queue
# Send work to the threads
$q->enqueue($_) for @ARGV;

# Worker threads
my $thread_limit = 8; 

my @thr = map {
    threads->create(sub {
        while (defined (my $item = $q->dequeue_nb())) {
            doStuff($item);
        }
    });
} 1..$thread_limit;
 
# terminate.
$_->join() for @thr;

    [download]
[reply]
[d/l]
Re^2: Perl Threads Boss/Worker Example
by locked_user sundialsvc4 (Abbot) on Apr 25, 2012 at 20:21 UTC

    Although I’m very glad for your Meditation and for your thought in posting it, I do think that Tanktalus does have a point worthy of its own meditation.   The number of work units to be done usually ought not to be equivalent to the number of threads to launch to do that work.   In the general case, that strategy does not scale well, and, when it “doesn’t scale,” it tends to “not scale well” rather badly.   A to-do-list based approach, on the other hand, much more closely matches what we tend to do in real life (except when senior management is in a panic...).   The number of workers is established according to the system’s ability to do work efficiently in parallel without interfering with one another by their mere presence.   The size of the to-do-list queue is both unpredictable and unrelated.   When 100 new customers pile into a fast-food restaurant, the establishment does not clone 100 new workers.

    Thanks for posting.

In Section Meditations