zeltus has asked for the wisdom of the Perl Monks concerning the following question:

Hi

I need, or perhaps more accurately, would like, to write some code whereby a parent would spawn off several child processes to work thru' a list of work-items.

fork to the rescue then. But I need to keep spawning off child processes up to a given max number of child processes as each one completes and whilst there are still work items on the list.

So, my question (at last!) - how could/should the parent process keep checking to see if there are less than the maximum number of child processes in action and thus be able to spawn off a new child process?

I have no code to show at this stage, but let us say, I want 5 child processes maximum hammering away at a list that contains 200 task items (I dunno, let's say, copying a specific MySQL table entry to one of several tables based on the value of the CustomerID column) and each task is expected to take a variable number of seconds.

This is more a design question than actual perl code but then, the solution will be perl, so that's my cop-out clause! :-)

I have been musing over the contents of perlipc but nothing there grabs me as being the One True Solution

As ever, any help gratefully received.

Bill

Replies are listed 'Best First'.
Re: Load-sharing parent
by pme (Monsignor) on Dec 08, 2014 at 16:39 UTC
Re: Load-sharing parent
by FloydATC (Deacon) on Dec 08, 2014 at 17:43 UTC

    The simplest way is to use a module as suggested, but if you want to invent your own wheel it's not very difficult. Simply have your master process keep the return value (pid) every time you fork(), use it as a hash key and regularly use waitpid() to check for finished child processes. The hash can be used for keeping track of whatever each child is tasked with. Cleaning up after a child could include reading back results before deleting the hash key.

    If you don't like the waitpid() looping you could also rely on a signal handler, but in practice I find this is a little more tricky to get 100% right.

    -- FloydATC

    Time flies when you don't know what you're doing