In order to effeciently use all the children, I want each child to only take one "task" from the pipe, leaving the next child to pickup the next "task". The work data is separated by a newline, and each child reads up to the first newline it encounters.

...

What would you do?

I thought that managing a pipe for each child would be unruly, but your example provided me with the information I needed. Specifically maintaining a hash of pipes associated to each child PID was what I was looking for. IO::Pipe provided the handle reference I needed to do that.

How could I have taken a reference to a FILEHANDLE like:

pipe( READER, WRITER );
$pipes{$pid} = [\*READER, \*WRITER];
[download]

Would that be the same thing as using IO::Pipe?

Do it the easy way:

#! perl -slw
use strict;
use threads;
use Thread::Queue;

$| = 1;

our $KIDS ||= 10;
our $WORK ||= 1000;
our $SLEEP||= 10;

sub kid {
    my( $Q ) = shift;
    my $tid = threads->tid;
    printf "Kid: %d started\n", $tid;

    ## Pick a work item off the queue and process it
    while( my $work = $Q->dequeue ) {
        printf "Kid: %d processing work item '%s'\n", $tid, $work;

        ## Replace the sleep with the code to process the work items
        sleep rand( $SLEEP );
    }
    printf "kid: %d ending\n", $tid;
}

## A queue for communications
my $Q = new Thread::Queue;

## Start the kids
my @kids = map{ threads->create( \&kid, $Q ) } 1 .. $KIDS;

## Wait till they're all up and running
sleep 1 until @{[ threads->list ]} == $KIDS;

## Feed the queue with work
## The limit just ensures we don't fill lots of memory
for my $workitem ( 1 .. $WORK ) {
    sleep 1 while $Q->pending > $KIDS *2;
    print "Queueing work item $workitem";
    $Q->enqueue( $workitem );
}

## Tell them to stop
$Q->enqueue( (undef) x $KIDS );

## And wait for them to do so.
$_->join for @kids;
[download]

Throw out the comments & demo print statements and it is all of 20 lines of code. It could easily be reduced to 15.

A test run

If it were my program (and if there were a reasonable number of child processes), I might instead use one pipe for every child, put them in a queue, and shift/push to send a task. Of course, that assumes that all of the jobs take about the same amount of time. If that's not the case, the child could write to the pipe when it finishes and only then could the parent push the pipe back on to the queue.

I don't think you can avoid some sort of locking if you have multiple processes contending for a single shared resource.

Note that this requires that your UDP packet doesn't get fragmented, I think.

And you can use the same technique with a single pipe. You need to make sure your writes don't get fragmented so you have to keep each task description under the "system buffer size", typically 4KB, I believe.

Don't use <$handle> read the tasks, use read or sysread instead. You can write the tasks with print if you use a single print statement and have auto flushing turned on or you can use syswrite to be more explicit about the atomic nature you desire.

- tye        

Correct. I think the OP implied that the requests were a line at a time. You are also correct about sysread/syswrite etc. It is actually quite simple, despite all the caveats.

but this sounds ineffiecient to me.

What would you do?

Semaphores, yes. My decision would proceed as:

1. Don't do this at all: I don't need such multiprocessing.
2. Have pipes for each child, with a protocol wherein children may request tasks and the parent may supply them with tasks. As long as I'm writing a protocol, it may as well also carry feedback from the tasks.
3. Use POE instead.
4. Use (not Perl:) Erlang instead.
5. Fail, colour the problem as insurmountable and backtrack within my overarching design.