Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hello Perl Monks,

I am using Thread::Pool to create a pool of threads (workers) (let's say 2) where my 10 processes can be queued and run in parallel, 2 at a time.

At some stage I want to abort the program. Usually some processes are finished, some are being processed and some are still pending to be processed in the pool.

What I do is to use the Thread::Pool->abort() method which, as I understood it, removes all pending processes and waits for all processed being processed right now. As soon as these are finished, then abort() returns.

Am I right to assume that at this stage, it is safe and proper to use Thread::Pool->shutdown() and then exit my program?

Unfortunately, when I call shutdown, I get a 'die' with the message:

Shutting down pool while there are still jobs to be done at /Library/Perl/5.10.0/Thread/Pool.pm (loaded on demand from offset 20056 for 2434 bytes) line 780

line 780 of Pool.pm is: 
die "Shutting down pool while there are still jobs to be done" 
if $belt->onbelt;

[download]

I am running: 

This is perl, v5.10.0 built for darwin-thread-multi-2leve

[download]

Below is an example code which demostrates the problem: 

#!/usr/bin/perl

use Thread::Pool;
use strict;
use warnings;

sub     my_process {
  print "Hi i am a spawned sub and I am sleeping. my ID is $_[0]\n";
  sleep(10); 
}
 
my $pool = Thread::Pool->new({
   workers => 2,
   maxjobs => 1,
   do => \&my_process
});
   
my ($i);
for($i=0;$i<10;$i++){ $pool->job($i); }

print "sleep for 5\n";
sleep(5);

print "will now abort\n";
$pool->abort();

print "will now shutdown\n";
$pool->shutdown();

print "all is normal but unfortunately i die before i come here\n";
exit(0);

[download]

Replies are listed 'Best First'.
Re: Thread::Pool shutdown dies after abort
by lidden (Curate) on Jun 29, 2011 at 13:11 UTC
    Why are you using both abort and shutdown?
    I think you are supposed to only use one of them depending on what you want. I do think the error message is confusing though.

    From the fine manual. 

       shutdown
        $pool->shutdown;

       The "shutdown" method waits for all jobs to be executed, remove
+s all
       worker threads, handles any results that still need to be strea
+med,
       before it returns.  Call the abort method if you do not want to
+ wait
       until all jobs have been executed.

       It is called automatically when the object is destroyed, unless
       specifically disabled by providing a false value with the
       "autoshutdown" field when creating the pool with new, or by cal
+ling
       the autoshutdown method.

   abort
       The "abort" method waits for all worker threads to finish their
       current job, removes all worker threads, before it returns.  Ca
+ll the
       shutdown method if you want to wait until all jobs have been do
+ne.

       You can restart the job handling process after calling "abort" 
+by
       adding workers again.

    [download]
[reply]
[d/l]
[select]
Re: Thread::Pool shutdown dies after abort
by dcmertens (Scribe) on Jun 29, 2011 at 13:20 UTC
    Hello anonymous monk. What you report here is not consistent with what I've seen. Thread::Pool looks like it depends on 5005threads, an old threading model that was deprecated with perl 5.8, I believe. Unfortunately, the Camel book was written for 5.6, so if you're basing your threading work on that, your code will be out of date. In fact, according to the Thread, the Thread module was removed from Perl in 5.10, so the module shouldn't even compile on your machine. I don't know too much about threads, but I'm pretty sure you can find similar (if not better) facilities for your work using the new threads interface. Good luck!
[reply]
      Thanks Honourable Monks, I will follow said advice and upgrade. temporarily i will use either abort or shutdown.
[reply]
Re: Thread::Pool shutdown dies after abort
by BrowserUk (Patriarch) on Jun 29, 2011 at 15:59 UTC

    A simple, working solution:

    #! perl -slw
use strict;
use threads;
use Thread::Queue;

$|++;

sub worker {
    my( $Q ) = @_;
    while( my $workitem = $Q->dequeue ) {
        print "Processing $workitem";
        sleep rand 10;
    }
}

our $THREADS //= 2;

my $Q = new Thread::Queue;

$SIG{'INT'} = sub{
    print "Sigint seen";
    $Q->dequeue while $Q->pending;
    $Q->enqueue( (undef) x $THREADS );
};

$Q->enqueue( <DATA> );

my @threads = map threads->new( \&worker, $Q ), 1 .. $THREADS;
$Q->enqueue( (undef) x $THREADS );
sleep 1 while $Q->pending;
$_->join for @threads;

__DATA__
WorkItem 1
WorkItem 2
WorkItem 3
WorkItem 4
WorkItem 5
WorkItem 6
WorkItem 7
WorkItem 8
WorkItem 9
WorkItem 10

    [download]
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
[d/l]

      I really like this approach, because it clearly delineates “the units of work that are to be processed” from “the pool of worker-bees that is responsible for processing them.”

      There should be enough threads/processes available to allow them to process the incoming requests expediently without unreasonable backlogs ... but they should process many requests during their lifetime, pulling them from a nice, thread-safe queue.   BrowserUK's solution clearly demonstrates just how easy that is.

      A fast-food restaurant uses exactly the same system to churn out hundreds of grease-burgers to five simultaneous lines in the lobby, as well as one or sometimes two drive-thru windows, using just a handful of grossly-underpaid employees.   And no workers are murdered by their manager during the entire shift.

        Thanks Monks for your answers. I think an approved thread pool package is needed with all the trimmings.
[reply]

Back to Seekers of Perl Wisdom