in reply to Fork multi processes

use Net::OpenSSH::Parallel: 
my $p = Net::OpenSSH::Parallel->new(workers => 5);
for my $host (@hosts) {
    $p->add_host($host);
}

$p->all(rsync_put => $src, $dst);
$p->run;

[download]

Replies are listed 'Best First'.
Re^2: Fork multi processes
by saurya1979 (Initiate) on May 04, 2012 at 08:26 UTC
    Thank you Salva. Is it possible to not use any external Perl module and use build-in modules to achieve it?
[reply]
      The entire point of Perl is that you do not have to. There are no points to be earned for laboriously doing what has already been done (better) by someone else, such that all you need to do to solve your problem is to install something and then write five or ten additional lines in order to do it.
[reply]
      Yes, sure, you can achieve it, though it would take you a while to do it.
[reply]
        Yes I know :-) and I am looking for the way to achieve it. So far I coded like below: 
        my @childs;
my @allfolders  => List of all folders

for ( my $i=0; $i<@allFolders; $i+=5 ) {

    my @sourceFolders = @allFolders[$i..$i+4];


        foreach my $folders ( @sourceFolders ) {

            my $pid = fork();

            if ( $pid ) {

                # parent
                push(@childs, $pid);
        
             } elsif ($pid == 0) {

        EXECUTe RSYNC HERE....
    
            } else {
                print "Error: Cannot fork process : $! \n",1);
            }
        }

        
    foreach ( @childs ) {
                waitpid($_, 0);
         }
   }

        [download]
        The problem here is that it executes rsync for 5 folders at a time and wait until all of them finishes. BUT, I want to start rsync for 6th folder immediately after 1 of 5 rsync is completed.
[reply]
[d/l]
      In my case, there is just 1 source and 1 destination server for rsync. I want to rsync directories located at different place on source server to the destination server. So I wanted to run multiple rsyncs to make it faster.
[reply]
        #!/usr/bin/perl

use strict;
use warnings;

use Net::OpenSSH::Parallel 0.12;
use Net::OpenSSH::Parallel::Constants qw(OSSH_ON_ERROR_IGNORE);

@ARGV == 3 or die <<EOU;
Usage:
  $0 host src_dir dst_dir

EOU

my ($host, $src, $dst) = @ARGV;

opendir my $dh, $src or die "unable to open $src";
my @ls = grep !/^\.{1,2}$/, readdir $dh;
close $dh;

my %now;
sub on_error {
    my ($pssh, $label, $error) = @_;
    warn "transfer of $now{$label} by $label failed\n";
    OSSH_ON_ERROR_IGNORE;
}

sub fetch_next {
    my ($pssh, $label) = @_;
    if (@ls) {
        my $file = shift @ls;
        $now{$label} = $file;
        warn "$label is copying $file\n";
        $pssh->push($label, rsync_put => "$src/$file", "$dst/$file");
        $pssh->push($label, sub => \&fetch_next);
    }
}

my $p = Net::OpenSSH::Parallel->new(on_error => \&on_error);

$p->add_host("worker$_", $host) for 1..5;
$p->all(sub => \&fetch_next);
$p->run;

        [download]
        Requires Net::OpenSSH::Parallel 0.12 I have just uploaded to CPAN.
[reply]
[d/l]

In Section Seekers of Perl Wisdom