yid has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks,

I've written a program that collects stats using system commands (SE Toolkit examples), re-formats the output, and sticks it into an RRD. However, it takes 10 seconds for each to run (since I'm collecting 5 second averages, & the first line of output is useless to me), and I don't want to wait 40 secs for each datapoint.

I'd like to run each of these commands in parallel, and I've looked at the Parallel:ForkManager module, but I don't think it applies since I want to run several different commands.

I also read about using the pipe function to get output back from a child process, but I need to do six at a time. I could just write 6 scipts, but that would mean an RRD for each script. Here's an example of what I'm running now (in serial):

sub Vmstat { $vmoutput_time = time; open (VMOUT, "$vmstat |"); $vmoutput = <VMOUT>; close (VMOUT); } sub Cpustat { $cpuoutput_time = time; open (CPUOUT, "$cpustat |"); @cpuoutput = <CPUOUT>; close (CPUOUT); } ... Vmstat () Cpustat ()
Is there a way to run multiple subroutines at once, and keep the output? Any assistance would be greatly appreciated.

Replies are listed 'Best First'.
Re: Running Subroutines in Parallel
by Dice (Beadle) on Mar 25, 2002 at 23:49 UTC
    I fear that my suggest might be too high-end for what you're really trying to do, but consider POE -- the Perl Object Environment. It is a framework for creation and management of co-operating multithreaded objects, managed by a kernel... all in Perl!

    Here are a few links to documentation on the subject...

    POE Central
    SourceForge page for POE
    A Beginner's Introduction to POE

    Good luck! (No, I'm not a POE expert, but I certainly want to be one one day...)

    Cheers,
    Richard

Re: Running Subroutines in Parallel
by zengargoyle (Deacon) on Mar 26, 2002 at 02:25 UTC

    Go for Event!

    It's ugly, unfactor'd, but works.

    use Event; use IO::File; my $f = IO::File->new('for t in 1 2 3 4;do echo one $t; sleep 4;done; +|'); my $g = IO::File->new('for t in 1 2 3 4;do echo two $t; sleep 4;done; +|'); my $h = IO::File->new('for t in 1 2 3 4;do echo three $t; sleep 4;done +; |'); my (@f,@g,@h); my $done; sub newIO { my ($f,$fr) = @_; Event->io( fd => $f, poll => 're', cb => sub { my $e = $_[0]; my $w = $e->w; my $fd = $w->fd; if ($fd->eof) { $pending--; $w->cancel; return; } my $line = <$fd>; push @{$fr}, $line; }, ); $pending++; } Event->idle( min => 1, max => 2, cb => sub { print "last one: $f[-1]"; print "last two: $g[-1]"; print "last three: $h[-1]"; Event::unloop if ($pending == 0); }, ); newIO($f,\@f); newIO($g,\@g); newIO($h,\@h); Event::loop; print @f; print @g; print @h;

    Gives:

    last one: one 1
    last two: two 1
    last three: three 1
    last one: one 1
    last two: two 1
    last three: three 1
    last one: one 1
    last two: two 1
    last three: three 1
    last one: one 2
    last two: two 2
    last three: three 2
    last one: one 2
    last two: two 2
    last three: three 2
    last one: one 2
    last two: two 2
    last three: three 2
    last one: one 2
    last two: two 2
    last three: three 2
    last one: one 3
    last two: two 3
    last three: three 3
    last one: one 3
    last two: two 3
    last three: three 3
    last one: one 3
    last two: two 3
    last three: three 3
    last one: one 3
    last two: two 3
    last three: three 3
    last one: one 4
    last two: two 4
    last three: three 4
    last one: one 4
    last two: two 4
    last three: three 4
    last one: one 4
    last two: two 4
    last three: three 4
    last one: one 4
    last two: two 4
    last three: three 4
    last one: one 4
    last two: two 4
    last three: three 4
    one 1
    one 2
    one 3
    one 4
    two 1
    two 2
    two 3
    two 4
    three 1
    three 2
    three 3
    three 4
    
Re: Running Subroutines in Parallel
by busunsl (Vicar) on Mar 26, 2002 at 06:41 UTC
    You could use the Coroutine Modules.

    I heard a talk about them at the German Perl Workshop, very impressive.

Re: Running Subroutines in Parallel
by webadept (Pilgrim) on Mar 26, 2002 at 01:16 UTC
    dice brought up Poe and I would have to suggest that there is your answer. However, while watching cartoons I thought that you could do a shell script and append your files. This only works on Linux type systems (maybe other Unix/BSD's as well)

    The idea is to have the shell script run all 4-6 of your scripts at the same time, the scripts would "append" data to the same file while running. something like this would be the shell script
    : perl umstat.pl & perl cpustat.pl &
    the "&" at the end starts a new PID for that program.

    Now that's about as weak an answer as anyone could give you.. unless we want to start a contest on "cheesy script writing", but it would work.

    Personally if this code means anything to you, then I would go with POE.

    Hope that helps.

    Glenn H.
Re: Running Subroutines in Parallel
by drifter (Scribe) on Mar 26, 2002 at 01:03 UTC
    I'm not sure if you can do this in this particular case, but fork() might do the trick. Ofcourse you'd need some magic to gather the output afterwards. Just an idea, why don't you open the both at the same time, eg:
    sub BothStat { $cpuoutput_time = time; $vmoutput_time = time; open (VMOUT, "$vmstat |"); open (CPUOUT, "$cpustat |"); @cpuoutput = <CPUOUT>; $vmoutput = <VMOUT>; close (CPUOUT); close (VMOUT); }
Re: Running Subroutines in Parallel
by tmiklas (Hermit) on Mar 26, 2002 at 01:45 UTC
    I've read suggestions mentioned above, and it's all rigt, but following the philosophy of Perl - you can do one thing in more than one way :-)
    You can try to set all the parts in 1 file... Fork them and gather the results... There are also some wired solutions (unsorted):
    1. sockets
    2. files with flock()
    3. pipes

    If you use sockets, then you write a server for them and the children are the clients that just send the results.
    Files with flock() are also some solution - in this case you have to read only from one handle.
    Pipes - almost the same as the sockets, only the loop differs and the data model is slightly different. When using sockets and pipes you would probably use some loop to skimm through open handles and check if there is something to read.

    If this script is to be simple as far as possible, i would use files with flock() i think.

    Greetz, Tom.
Re: Running Subroutines in Parallel
by Dinosaur (Beadle) on Mar 26, 2002 at 17:57 UTC
    I'm not positive that I understand what you're trying to do (e.g., what's an RRD?) -- but assuming that I'm close, I'd propose a low-tech solution. Start all the system commands simultaneously and pipe their output back to you:
    open(VMSTAT, "vmstat |"); open(CPUSTAT, "cpustat |"); open(...
    Then use select (the 4-element version, not the output file select) to let you know when to read each of the pipes. You'll end up reading them in random order (depending on the timing of each command producing its output), but if that matters, you can collect everything and output your results when you've got them all.

    Select is a bit cumbersome to use (good example in the Camel book). Perhaps you get the parallelism you need just by starting the commands in parallel. Then just read from them one at a time. Do you need to read just one line from each command, or are you expecting multiple lines?

    -- Dino

Re: Running Subroutines in Parallel
by yid (Initiate) on Mar 28, 2002 at 18:56 UTC

    Thank you all for your replies. I am going to look at POE and Coroutine in the future, because it looks like they would both solve this problem. However, I've found a alternative to running all of these commands that serves me better.

    I'm going to use the Sun::Solaris::Kstat module to pull all of these stats, and update at the same time. So it'll take a matter of nanoseconds to get all the stats I need every five seconds instead of having to wait on shell commands.

    It'll be pretty sweet when its done. Look for it on freshmeat!