Simplistically:

#! perl -slw
use strict;
use threads;
use Thread::Queue;
use DBI;


our $N //= 4;

sub child {
    my $Q = shift;
    while( my $work = $Q->dequeue ) {
        for my $thing ( @{ $work } ) {
            ## process data items
        }
    }
}

my $Q = new Thread::Queue;

my @kids = map threads->create( \&child, $Q ), 1 .. $N;

my $dbh = DBI->connect( ... );
my $sth->prepare( ...SQL HERE... );

while( my $ref = sth->fetch_* ) {
    $Q->enqueue( share_clone( $ref ) );
}

### Assume exiting the loop above means no more data
$dbh->disconnect;

## tell the kids to die
$Q->enqueue( ( undef ) x $N );

## And wait for kids to clean up
$_->join for @kids;;
[download]

Wow, I wish I would of found this thread before I hacked around with Parallel fork manager, Thanks for this Example

Hi,
I am getting the following error when I tried to throw an error from my child thread.

Error: Reached EOF after 0 bytes, header is minimum 4 bytes
Operation """": no method found, argument in overloaded package main at /home/jpotter/testscript.pl

Any Idea what the above error is?
Thanks!

Any Idea what the above error is?

None whatsoever.

Reinforcing what was just suggested (in code) ...

The main thread could be the one that gathers the work-requests, but it shouldn’t try to figure out which of the various child-threads might be the one that gets to do the next one.   (That is a race-condition, and unnecessary.)  

Simply place the new work-request onto a FIFO (first-in first-out) queue that n child processes are simultaneously listening on.   You know that one of them will get the message; you don’t know, and don’t care, which one.

The child threads really don’t care too much about the parent process, nor about their other siblings, nor about how many siblings there might be.   They simply:   wait for a request to arrive, carry it out, and send some kind of notification (through another queue...) that they have done so.   They do this until they are finally told (say, by another message...) to terminate.

In quite a few applications I have built and witnessed, the main thread basically has nothing to do, except maybe to keep the user entertained.   Everything is done by children:   one child accepts requests, n brothers and sisters do the work, and one child sends the responses back.   All the moving parts are stitched together using FIFO queues.

1. Main thread/process has to create a pre determined set of threads. Say 5 child threads --Can I create n no of threads in a forloop?

If you can create one of something, you can always create n of those things in a for loop.

2. Main process constantly queries the database and picks a piece of work and look for a ready child thread to pick it up and assign it to thread if it is available or wait for a ready thread.

Luckily, you don't have to worry about too many of those details. perl provides a queue (via Thread::Queue) which you can add data to, and whatever thread is ready will grab the data from the queue.

Basically, you create n threads and have them read from the same empty Q. When a Q is empty, a thread blocks until it can extract data from the Q. With that setup, the threads will wait like hungry wolves for you to add data to the Q. A Thread::Queue is also thread safe, which means that only one thread will end up reading a specific piece of data.

Then all you have to do is add data to the Q, and the threads will pounce on it. If all the threads are busy, then the data will remain in the Q until a thread is ready to "eat" again. You make the threads repeatedly eat from the Q by using a while loop in your thread: inside the loop you read from the Q and process the data.

3. Continue the process until all work is done and detach all child threads.

The only trick is signaling when you are done adding data to the Q, so that the threads can stop trying to read data from the Q and terminate. You tell the threads you don't need them anymore by adding data to the Q that signals the threads that they should stop trying to read from the Q. The threads should test the value of the data they read from the Q before processing the data. For the signal, you can use value undef, the string "THE END", or anything else you want.

Add one signal to the Q for every thread there is. When a thread reads the signal, the while loop that reads the data will terminate, so a thread can never read two signals. And if a thread cannot read two signals, and there is one signal for every thread, then every thread will end up reading one signal and terminate. Just make sure whatever value you use as the signal cannot possibly be valid data.

Thanks much Everyone for the quick threads tutorial! It really helped me.

Anytime you are working with Perl and therefore with CPAN, you seriously need to be sure that you are “taking the long view.”   No matter what you are doing, it has probably been done before, such that there are numerous existing frameworks and components that can be “dropped into” your new project.   In other words, you are probably not building “a multi-threaded program.”   You’re building “a server,” and perhaps, one intended to “do this-or-that” such that “this-or-that” is some well-known and well-understood task.   Given this reality, you are probably in yet another “build vs. buy decision” that will be resolved (as most such things are) in favor of “buy” (albeit in this case, “for free”).

This is exactly why companies make a good living selling, say, prefabricated roof-joists ... pre-hung doors ... even manufactured walls and houses.   These are things that you could build for yourself, even to the point of chopping down your own trees and forging your own square nails, but it is highly unlikely to make practical sense to do so unless you are a historical preservationist.

Needless to say, no one can do this research and analysis for you, and “There Are No Silver Bullets.™”   But there are plenty of very excellent pre-hung doors and windows to choose from.

For instance:   Do you want a high-performance web server that can support a very large number of clients?   That is an extremely well-known and well-understood task.   Therefore, accomplishing that task can be (almost...) as simple as:   Some::Server::Class->new->run;   Seriously.   (You get my drift...   You will spend “80% of your time in 20% of the code,” so begin by grabbing “the other 80% of the code” from a carefully selected CPAN shelf.)

“Well, you can run the whole distance if you want to, o’course, but why don’t we just drive up to the finish line, and you can stroll across from there?”

No one has mentioned using threads shared variables, and the fact that you can share filehandles between the main thread and children by passing the fileno of the filehandle. See Simple threaded chat server for an example of this.