rahulruns has asked for the wisdom of the Perl Monks concerning the following question:

I have two PERL SCRIPT that stress CPU and MEMORY. I want to run them in parallel on hosts so I am trying to copy them on all the hosts and run them in parallel. I am utilizing a module Net::OpenSSH::Parallel to copy (which is working) and run the commands on all the nodes. I am not able to run these two SCRIPTS in parallel. I have tried to take the SCRIPT to background but only one of them runs. Could you guide where am I going wrong?

CPU SCRIPT #!/usr/bin/perl use strict; use warnings; my @cpu_cores = qw(4); while (--$cpu_cores[0] and fork) {}; while () {}; MEMORY SCRIPT #!/usr/bin/perl use strict; use warnings; print "$$"; fork for 1 .. 4; for (1 .. 10000) { my $a = "xxxxx" x int rand 10_000_000; my $b = ~$a; my $c = reverse $b; MAIN SCRIPT use strict; use warnings; use Net::OpenSSH::Parallel; use Term::ANSIColor; use DBI; #-------------------------------------------------------- #FINDING HOSTS FROM CLOUDERA MANAGER DATABASE #-------------------------------------------------------- print color 'bold blue'; print " Finding hostname of the hosts configured\n\n"; my $ip_address = `ping r01mgt -w 1 | awk -F "(" '/PING/{print \$2}' | +awk -F")" '{print \$1}'`; chomp($ip_address); # connect my $dbh = DBI->connect("DBI:Pg:dbname=xxxx;host=$ip_address", "xxxx", +"xxxx", {'RaiseError' => 1}); # execute SELECT query my $sth = $dbh->prepare("SELECT name FROM hosts"); $sth->execute(); # iterate through resultset and create an array my @hosts; while(my $ref = $sth->fetchrow_hashref()) { my($nodename, $hadoopname) = $ref->{'name'} =~ m/(\w+).(\w+)/; push(@hosts, "$nodename"); } @hosts = sort (@hosts); print color 'reset'; #----------------------------------------------------------- #COPYING THE PERL SCRIPTS AND EXECUTING THEM #----------------------------------------------------------- my $pssh = Net::OpenSSH::Parallel->new(); $pssh->add_host($_) for @hosts; $pssh->push('*', scp_put => '/root/cpu.pl', '/root/'); $pssh->push('*', scp_put => '/root/memory.pl', '/root/'); $pssh->push('*', command => '/usr/bin/perl', '/root/cpu.pl', '/tmp/cpu +'); $pssh->push('*', command => '/usr/bin/perl', '/root/memory.pl', '/tmp/ +memory'); $pssh->run; USING THREADS TO RUN SCRIPTS IN PARALLEL #!/usr/bin/perl use strict; use warnings; use threads; sub cpu { my @cpu_cores = qw(4); print "CPU"; while (--$cpu_cores[0] and fork) {}; while () {}; } sub memory { print "MEMORY"; fork for 1 .. 4; for (1 .. 10000) { my $a = "xxxxx" x int rand 10_000_000; my $b = ~$a; my $c = reverse $b; } } my $firstThread = threads->create(\&cpu); my $secondThread = threads->create(\&memory); $_->join() foreach ( $firstThread, $secondThread );

Replies are listed 'Best First'.
Re: Running PERL SCRIPTS in Parallel
by salva (Canon) on May 22, 2013 at 07:42 UTC
    Net::OpenSSH::Parallel does not support parallelization inside the host (it's on the TODO list). Though, for simple cases it can be done using the parsub action:
    # untested! require POSIX; sub ssh_run_parallel { my ($label, $ssh, @cmds) = @_; my @pids = map $ssh->spawn(@$_), @cmds; my $error = 0; for (@pids) { waitpid $_, 0; $error ||= $?; } POSIX::_exit($error); } my $pssh = Net::OpenSSH::Parallel->new(); $pssh->add_host($_) for @hosts; $pssh->push('*', scp_put => '/root/cpu.pl', '/root/memory.pl', '/root/ +'); $pssh->push('*', parsub => \&ssh_run_parallel, ['/usr/bin/perl', '/root/cpu.pl', '/tmp/cpu'], ['/usr/bin/perl', '/root/memory.pl', '/tmp/memory']); $pssh->run;

    update: the error pointed by rahulruns below has been fixed!

      I tried to run the code but was not able to understand my @pids = map $ssh->spawn(@$_) for @cmds; It is giving error Not enough arguments for map at stress.pl line 44, near ") for " Execution of stress.pl aborted due to compilation errors. I am not able to troubleshoot. Could anyone help

      my @pids = map $ssh->spawn(@$_) for @cmds; Error Not enough arguments for map at stress.pl line 44, near ") for " Execution of stress.pl aborted due to compilation errors.
        I have corrected the code.
        my @pids = map $ssh->spawn(@$_), @cmds;
        is equivalent to ...
        my @pids; for my $cmd (@cmds) { push @pids, $ssh->spawn(@$cmd); }
        It uses the Net::OpenSSH object passed to the sub to start all the commands in parallel.

        Thank You for the Fix. I too was able to troubleshoot

        my @pids = map $ssh->spawn(@$_) @cmds;
Re: Running PERL SCRIPTS in Parallel
by walkingthecow (Friar) on May 22, 2013 at 06:42 UTC
Re: Running PERL SCRIPTS in Parallel
by QM (Parson) on May 23, 2013 at 08:03 UTC
    Not sure if this helps you, given all of the excellent comments so far...

    I recently posted an example using Parallel:ForkManager (here).

    I only had to setup ssh w/o passwords to get this to work.

    -QM
    --
    Quantum Mechanics: The dreams stuff is made of