How do I queue perl subroutines to a thread queue instead of data?

static0verdrive has asked for the wisdom of the Perl Monks concerning the following question:

Background: In reading how to multithread my perl script, I read (from http://perldoc.perl.org/threads.html#BUGS-AND-LIMITATIONS)

On most systems, frequent and continual creation and destruction of threads can lead to ever-increasing growth in the memory footprint of the Perl interpreter. While it is simple to just launch threads and then ->join() or ->detach() them, for long-lived applications, it is better to maintain a pool of threads, and to reuse them for the work needed, using queues to notify threads of pending work.

My script will be long-lived; it's an PKI LDAP directory monitoring daemon that will always be running. The enterprise monitoring solution will generate an alarm if it stops running for any reason. My script will check that I can reach another PKI LDAP directory, as well as validate revocation lists on both.

Problem: Everything I can find on google shows passing variables (e.g. scalars) to the thread queue rather than the subroutine itself... I think I'm just not understanding how to implement a thread queue properly compared to how you implement a thread (without queues).

Question 1: How can I "maintain a pool of threads" to avoid the perl interpreter from slowly eating up more and more memory?
Question 2: (Unrelated but while I have this code posted) Is there a safe amount of sleep (or a better method than sleep) at the end of the main program so that I don't start a thread more than once in a minute? 60 seems obvious but could that ever cause it to run more than once if the loop is fast, or perhaps miss a minute because of processing time or something?

Thanks in advance!

#!/usr/bin/perl

use feature ":5.10";
use warnings;
use strict;
use threads;
use Proc::Daemon;


### Global Variables
use constant false => 0;
use constant true  => 1;
my $app = $0;
my $continue = true;
$SIG{TERM} = sub { $continue = false };

# Directory Server Agent (DSA) info
my @ListOfDSAs = (
    { name => "Myself (inbound)",
      host => "ldap.myco.ca",
      base => "ou=mydir,o=myco,c=ca",
    },
    { name => "Company 2",
      host => "ldap.comp2.ca",
      base => "ou=their-dir,o=comp2,c=ca",
    }
);    


### Subroutines

sub checkConnections
{   # runs every 5 minutes
    my (@DSAs, $logfile) = @_;
    # Code to ldapsearch
    threads->detach();
}

sub validateRevocationLists
{   # runs every hour on minue xx:55
    my (@DSAs, $logfile) = @_;
    # Code to validate CRLs haven't expired, etc
    threads->detach();
}



### Main program
Proc::Daemon::Init;

while ($continue)
{
    my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localti
+me(time);

    # Question 1: Queues??

    if ($min % 5 == 0 || $min == 0)
        { threads->create(&checkConnections, @ListOfDSAs, "/var/connec
+t.log"); }

    if ($min % 55 == 0)
        { threads->create(&validateRevocationLists, @ListOfDSAs, "/var
+/RLs.log"); }

    sleep 60; # Question 2: Safer/better way to prevent multiple threa
+ds being started for same check in one matching minute?
}

# TERM RECEIVED
exit 0;
__END__
[download]

Comment on How do I queue perl subroutines to a thread queue instead of data? Download Code

Replies are listed 'Best First'.
Re: How do I queue perl subroutines to a thread queue instead of data? by Random_Walk (Prior) on Apr 25, 2013 at 14:03 UTC
If I just had two threads to signal I would create a queue for each one and just enqueue the messages when I felt the need. the thread can then block on a queue read and do nothing until it gets a message Updated to mention threads->create `... use Thread::Queue; my $conn_q = thread::queue->new() my $rev_q = thread::queue->new() sub connection_check { while (my $msg = $conn_q->dequeue) { # will block 'til it gets sign +aled # Do something } } ... # in main $conn_thread = threads->create('connection_check'); $conn_q->enqueue("Get on with it");` [download] `$min % 55` will be zero at 55 and at 0 minutes. Perhaps two timers would do your trick `my %timer = ( check_conn => 0, check_rev => 0, }; my %interval = ( check_conn => 5, # check conn every 5 min check_rev => 55, # and revocation at 55min }; while (1) { my $now = time; if ($now + interval{check_conn} * 60 > $timmer{check_conn} ) { #pa +ssed mark $timer{check_conn} = $now + interval{check_conn} * 60; # signal conn thread to do its work $conn_q->enqueue("Get on with it"); } # similar for check rev list # sleep 10; # or 60, or 300... }` [download] Cheers, R. Pereant, qui ante nos nostra dixerunt!	[reply] [d/l] [select]
Re^2: How do I queue perl subroutines to a thread queue instead of data? by static0verdrive (Initiate) on Apr 25, 2013 at 14:43 UTC
Thanks for your reply! OK I think I'm starting to get it... are you saying that I never thread->create but instead only $conn_q->enqueue in the #main? In other words, my $conc_q = thread::queue->new() would exist at the top and no thread->create would exist anywhere because the enqueue replaces it?	[reply]
Re^3: How do I queue perl subroutines to a thread queue instead of data? by Random_Walk (Prior) on Apr 25, 2013 at 15:06 UTC
No Sorry! you still need to `thread->create`! Just make a thread that listens in the queue with the `while (my $msg = $q->dequeue) { ...` and it will wait until signalled. You can just use this to hold it up or you can check the contents of `$msg` and behave accordingly Cheers, R. Pereant, qui ante nos nostra dixerunt!	[reply] [d/l] [select]
Re^4: How do I queue perl subroutines to a thread queue instead of data? by static0verdrive (Initiate) on Apr 25, 2013 at 15:14 UTC
Re: How do I queue perl subroutines to a thread queue instead of data? by locked_user sundialsvc4 (Abbot) on Apr 26, 2013 at 01:46 UTC
The key design-thought to keep in mind ... in any language, in any situation ... is that “a thread or process is a hamburger cook, not a single hamburger.” Each thread goes through the same life-cycle as does any cook in a burger-joint: wait for an order, cook it, deliver it, and then wait for another one. And, at the end of the shift, he comes out alive. Maybe he cooked two hamburgers that day ... maybe he cooked a hundred. Often, threads are specialists ... the “burger-meister,” the “fry guy” and so on. They sit at one station, and listen to one queue. Or they can be generalists, all grabbing from one queue and doing whatever comes. Perl provides plenty of reliable “thread-safe queues,” and even complete workload-management systems like POE. Your choice. But, whatever it is, it ought to make sense if mapped to a real-world situation like a fast food restaurant. Don’t ask the operating system’s thread/process dispatcher to dispatch your application’s work flow, because it can’t. The so-called “flaming arrow approach” does not work in practice. A pool of specialized or generalized worker-bees carry out the work, and live to tell the tale. It is certainly a viable approach to structure the system so that Perl Objects are placed on the queue and the only thing that the worker-bee actually does is to call some `execute()` method implemented (in the abstract ...) by the base class.

Updated to mention threads->create