How to send & receive data to many child processes?

vancetech has asked for the wisdom of the Perl Monks concerning the following question:

I am writing a perl daemon for multiple servers that will process time consuming tasks from a single mysql database. The server maintains a connection to the database gathering tasks when they are available.

Pre-forked children report to the parent process via a PIPE telling of their status 'idle' or 'busy' so the server knows which child to assign the next task to.

The task data is usually less than 128bytes and there are 20 or more child processes at anyone time.

What would be the best way of delegating the task to the children?

I've heard of using shared memory, having each child access a specific portion for their assignments.
I've heard of opening a bi-directional PIPE to the child, but unsure if managing 20+ PIPEs is effective.
I've had a suggestion that the children should access the database for the tasks. If the daemon is run on 5 seperate boxes there would be 5*20 = 100 processes querying the database endlessly. That doesn't sound effecient to me.

What would be an appropriate solution?

Thankyou Monks!

Comment on How to send & receive data to many child processes?

Replies are listed 'Best First'.
Re: How to send & receive data to many child processes? (one pipe) by tye (Sage) on Mar 10, 2006 at 07:58 UTC
I'd just use a single pipe for sending out the tasks. Each child does a (blocking) read on the pipe when it is ready for the next task and exactly one child will get it (since the task data is smaller than the "system buffer size" -- likely 4KB). That way the parent process doesn't have to do any selecting or managing; it simply writes each task to the pipe. If you want to track the status of each child as well, I'd just have a second pipe that the children write messages to, including their own PID. To make things simple, I'd use one process for writing the tasks to the first pipe and another for reading status from the second pipe. Then no non-blocking I/O is needed. I'd actually use two named pipes so that spawning new worker children or replacing either of the two manager processes becomes trivial. - tye	[reply]
Re^2: How to send & receive data to many child processes? (one pipe) by vancetech (Beadle) on Mar 10, 2006 at 08:32 UTC
Yah perfect, that works as expected. Thanks tye!	[reply]
Re^2: How to send & receive data to many child processes? (one pipe) by perrin (Chancellor) on Mar 10, 2006 at 18:27 UTC
Nice idea and explanation, tye. Thanks.	[reply]
Re: How to send & receive data to many child processes? by samtregar (Abbot) on Mar 10, 2006 at 02:59 UTC
Using bi-directional pipes should work fine. You'll need to be careful to avoid deadlock, but otherwise it should be fairly straight-forward. I just finished a small project which uses Parallel::ForkManager, IO::Pipe and IO::Select to farm out SMTP-sending jobs to a pool of sub-processes. It seems to be working well so far. -sam	[reply]
Re^2: How to send & receive data to many child processes? by vancetech (Beadle) on Mar 10, 2006 at 04:00 UTC
thank you Sam... however I'm confused about how to fork and manage multiple pipes specific to each process... can you find any code to illustrate this to me?	[reply]