locked_user sundialsvc4 has asked for the wisdom of the Perl Monks concerning the following question:

Okay, I’m working with AIX and Perl 10.5.0.   And what I need, very quick-n-easy, is a program that can spawn n child processes, snooze until one of them finishes, gather status, update a database, launch another child.   (These are all processes, each of which might run for five hours straight.)

The main bit of business is ... I really don’t want to have to think too much about how to launch, watch, and wait-for child processes.   I’d really love to be able to start out tomorrow morning by picking a CPAN module, spend the morning configuring the thing (and building-out the rest of it, very minimally) and I am limping along (at least) by tomorrow afternoon.   Things like “update a database” are not my primary concern here.   Watching a (small-n) number of child-processes for hours on end and “doin’ all the usual schtick without having to code much of it” ... is.

Basically, we have been trying to use a “task spooler” program, ts, and the thing just isn’t working reliably.   It “falls dead,” although the spawned processes continue to survive out there (for a while).   People spend hours watching them, and updating Excel spreadsheets by hand.   aargh...   Let’s just say that I do not want to find myself obliged to become one of those people.   And of course, the misbehavior of ts, which was never an ideal arrangement, is a running disaster just waiting to be stomped flat by a Camel.

Quick ’n easy process-watching, environment=AIX, yours-truly=very experienced.   Main goal=lazy and quick, plug the dike.   Off-the-cuff suggestions?

Replies are listed 'Best First'.
Re: Bein' lazy: need a child-process monitor, quick 'n fast
by BrowserUk (Patriarch) on Oct 19, 2010 at 02:57 UTC

    On the basis of your description, Parallel::ForkManager might be appropriate.

    You could DIY-it in a few lines of Perl using threads or fork very easily, but I think the above package is probably what you're looking for.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
Re: Bein' lazy: need a child-process monitor, quick 'n fast
by zentara (Cardinal) on Oct 19, 2010 at 16:30 UTC
    Off-the-cuff suggestions?

    Here are couple of quick and dirty fork managers by 2 of the more illustrious monks.

    #!/usr/bin/perl #by merlyn use POSIX ":sys_wait_h"; my @tasks = (1..398); my %kids; { while (@tasks and keys %kids < 5) { $kids{fork_a_task(shift @tasks)} = "active"; } { my $pid = waitpid(-1, 0); if ($pid == -1) { %kids = (); } else { delete $kids{$pid}; } } redo if @tasks or %kids; } sub fork_a_task { my $i = shift; my $pid = fork; return $pid if $pid; unless (defined $pid) { warn "cannot fork: $!"; return 0; } ## do stuff for task $i goes here... print "Doing $i\n"; exit 0; }
    and
    #!/usr/bin/perl #by Abigail of perlmonks.org #Some times you have a need to fork of several children, but you want +to #limit the maximum number of children that are alive at one time. Here #are two little subroutines that might help you, mfork and afork. They + are very similar. #They take three arguments, #and differ in the first argument. For mfork, the first #argument is a number, indicating how many children should be forked. +For #afork, the first argument is an array - a child will be #forked for each array element. The second argument indicates the maxi +mum #number of children that may be alive at one time. The third argument +is a #code reference; this is the code that will be executed by the child. +One #argument will be given to this code fragment; for mfork it will be an + increasing number, #starting at one. Each next child gets the next number. For afork, the + array element is #passed. Note that this code will assume no other children will be spa +wned, #and that $SIG {CHLD} hasn't been set to IGNORE. mfork (10,10,\&hello); sub hello{print "hello world\n";} print "all done now\n"; ################################################### sub mfork ($$&) { my ($count, $max, $code) = @_; foreach my $c (1 .. $count) { wait unless $c <= $max; die "Fork failed: $!\n" unless defined (my $pid = fork); exit $code -> ($c) unless $pid; } 1 until -1 == wait; } ################################################## sub afork (\@$&) { my ($data, $max, $code) = @_; my $c = 0; foreach my $data (@$data) { wait unless ++ $c <= $max; die "Fork failed: $!\n" unless defined (my $pid = fork); exit $code -> ($data) unless $pid; } 1 until -1 == wait; } #####################################################

    I'm not really a human, but I play one on earth.
    Old Perl Programmer Haiku ................... flash japh
Re: Bein' lazy: need a child-process monitor, quick 'n fast
by aquarium (Curate) on Oct 19, 2010 at 03:41 UTC
    you could possibly use net::daemon or other. an outline of how to write a preforking daemon http://www.petercooper.co.uk/archives/000702.html. that talks about http daemon, just take the prefork bits you want. you can make your starting daemon / parent start from init.d to make it resilient.
    maybe if you elaborate a bit more about the scheduling requirements, someone may be able to offer more help.
    i don't have any understanding of what your pre-forked processed are supposed to do..but a number of pre-forked processes that run for hours doesn't sound like a reasonable setup to start with. perhaps this setup could benefit from a shift in approach? otherwise just run a cluster all on the same box or distributed?
    the hardest line to type correctly is: stty erase ^H
Re: Bein' lazy: need a child-process monitor, quick 'n fast
by locked_user sundialsvc4 (Abbot) on Oct 19, 2010 at 18:00 UTC

    These suggestions seem to be quite satisfactory.   Many thanks.

    To answer the question of “scheduling:   “ at the moment, there isn’t any.”   The Perl program will simply launch n children and wait (sometimes hours) for any one of them to finish, then start the next one on a prescribed list (table).   (Stupidly simple, and absolutely essential.   Maybe someday it will get fancier, but then again, ts never was.)   This should “plug the leak in the dike,” and that’s the only thing I need The Camel to do now.   To replace an unreliable, unknown utility with a reliable beast of burden (having a large, friendly humph).