ktelep has asked for the wisdom of the Perl Monks concerning the following question:

I'm going to be doing a system copy from one box to another via 6 fiber channel links (we're not talking your daddy's PC here)

What i'd like to do is kick off a bunch of rcp's on the unix shell concurrently. This is all fine and dandy, but i'd like to have them all log to the same logfile. So I was thinking about forking them all off of from a perl script, and piping their STDOUT's back to the parent and then just having teh one script write to the logfile to avoid collisions....

Any ideas on how to startup all these jobs and pipe STDOUT back to the parent?

Replies are listed 'Best First'.
Re: Multple Processes
by chromatic (Archbishop) on Feb 16, 2001 at 21:57 UTC
    I'd probably just pass in the log file name, open it for exclusive append writing access with sysopen and flock, and sleep in the children waiting for an exclusive lock.

    It depends on how many kids you have, what the system load is like, and how long the jobs take to process if this is a workable solution.

    It seems conceptually easier to me than trying to coordinate lots of children trying to write to the same pipe in the parent, but that's just me.

Re: Multple Processes
by zigster (Hermit) on Feb 16, 2001 at 22:16 UTC
    Use select to multiplex over many reading file handles (see code below butchered from the cook book). Checkout the following nodes for more info:
    • select for info about the older non OO version of select
    • IO::Select for info about the OO version listed below
    • perlman:IPC::Semaphore and IPC::SysV if you want to implement some kind of process locking (you hint that you do but should not have to with select)
    • Here For a comprehensive treatment of the use of select. (Old non OO stylee)
    • Look at Section 7.13 of the cook book for the original reference to the code mapped out below
    use IO::Select; open FHANDA,"processA|" || die(".."); open FHANDB,"processA|" || die(".."); open FHANDC,"processA|" || die(".."); $Select = IO::Select->new(); $Select->add(*FHANDA); $Select->add(*FHANDB); $Select->add(*FHANDC); if (@ready = $slect->can_read(0)) { for (@ready) { #Process the file handle (do the read and write) } }
    Update Just a thought but in the reaper sub (not written above) it would be nice to $Select->remove() the file handles as the processes die.
    --

    Zigster
Re: Multple Processes
by TheoPetersen (Priest) on Feb 16, 2001 at 23:13 UTC
    My immediate thought was to reach for Expect, which I use for managing process from Perl in other scripts. Seems like a bit of overkill just to merge STDOUT, but it does work, since (by default) the output of commands spawned by Expect goes to the script's STDOUT.

    Here's a sample script, with debugging lines left in for flavor:

    #!/usr/bin/perl -w use strict; use Expect; $Expect::Multiline_Matching = 0; my @todo = ("ls /etc", "ps axw", "du"); my @cmds; foreach my $cmd (@todo) { print "spawning $cmd\n"; push @cmds, Expect->spawn($cmd); die "Didn't spawn $cmd\n" unless $cmds[-1]; } print "cmds = @cmds\nExpecting...\n"; while (@cmds) { expect( 1, '-i', \@cmds, [ qr"^.*\n", sub {print "\n", shift, " did something\n"; exp_continue;} ], [ 'eof', sub { my $done = shift; print "$done is finished\n"; @cmds = grep {$_ != $done} @cmds; print "cmds = @cmds\n"; } ], ); } print "Everything finished\n";
    The first pattern match is useless, it just adds some output to say which command was active. The second match is the only required one for your purpose. It deletes commands from the list as they finish, and recycles the expect call.

    One benefit of going this route is that you could put in pattern matches for rsh error output and take special action, while ignoring most of the output.

    Expect is nice for letting you take care of things that matter and ignore the bulk of a process's output.