use Net::OpenSSH;
my $ssh = Net::OpenSSH->new($myhostname, user => $user,
master_stderr_discard => 1 # quiet!
);
my $output = $ssh->capture("cat /etc/redhat-release");
| [reply]
[d/l] |
Yes there are some Secure Shell modules that do provide some multi-server operations. However, many of these modules can be difficult to compile on various platforms, require extensive modules like the crypt-style modules, only address parallel operations when running the remote command, do not easily account for hung Secure Shell sessions, etc... Moreover, what about pre and post processing of the information? Such post processing cases, all information in returned to the central point of execution and typically become single threaded--loss of efficiency. Respectfully this may be what you want, or you can have each thread perform its portion of the post process as part of its responsibility. Expand this thought to pre processing, what about checking if the server is pingable or if the name provided is in DNS? These are just a couple of examples of typical issues at hand.
The example provided accounts for a more robust solution, which allows for one to extend to suit their design goals. Also, this solution can easily be extended to Secure Copy, as this would just be another command to run.
Finally if you look at the 'SSH_OPT' which I have defined, these options can also be used with in shell-based scripts, as an alias in the user environment, etc... Such options should almost always be used with a batch style Secure Shell script. It accounts for connectivity time out, known_host issues, remote agent death, and more.
--Poet
| [reply] |
| [reply]
[d/l] |
Your comment talks about SSH modules as if they were all the same but they are not. There are several of them and every one has its own strengths and weaknesses and none is better than the others in all the matters.
The example provided accounts for a more robust solution, which allows for one to extend to suit their design goals
The piece of code you have posted suffer from the same problem, it may suit your needs but it is not the solution to all Perl & SSH tasks.
It has several limitations: no connection reuse, no password authentication, no passphrases, no synchronization between different workers, no command quoting, usage of highly insecure options, wrong handling of hung sessions, etc.
Also, this solution can easily be extended to Secure Copy, as this would just be another command to run.
Almost any SSH module available from CPAN already supports SCP and/or SFTP out of the box.
Really, you should get more familiar with the SSH modules available from CPAN, specially the new ones. They are better than what you think!
| [reply] |
As the maintainer of Net::SSH, let me be the first to encourage you to look at alternate modules. Net::SSH was useful when written but is very much deprecated legacy code these days.
Net::OpenSSH looks like the best modern choice if you want to use the external ssh binary. Other alternatives worth noting are Net::SSH::Perl (perl reimplementation of the protocol) and Net::SSH2 (uses libssh2). | [reply] |
Thanks for all the responses.
I'll take a look at Net::OpenSSH.
When I started this project I was using Net::SSH::Perl, this module works fine, but is far too slow for my needs. I need to get info from several hundred servers. Using Net::SSH::Perl was taking several hours vs. ~20 minutes with Net::SSH.
| [reply] |
Net::OpenSSH is awesome!
With it's ability to reuse ssh connections it cut my average run time by 50%.
(And one particularly inefficient function was reduced from 14 minutes to 4).
IMHO, well worth the price of admission.
| [reply] |
sub sshopen2 {
my($host, $reader, $writer, @command) = @_;
@ssh_options = &_ssh_options unless @ssh_options;
open2($reader, $writer, $ssh, @ssh_options, $host, @command);
}
i.e., the $ssh, @ssh_options, $host, @command constitutes the ssh command that's being run. @ssh_options doesn't seem to be accessible via the module's API (from what I can tell), but you could try to pass your -q option as the first element of the @command array. This should work because ssh accepts most options also after its host argument.
Alternatively, patch the module's _ssh_options routine to return the desired options... (Or even don't use the module at all, and simply run the ssh command yourself with open2)
| [reply]
[d/l]
[select] |
My brother some of what you describe is exactly why I do not use those modules. PLEASE check this out and adapt as needed. The communication and timer aspects of this are mine, however the threading efficiency belongs to BrowserUk... and that for that I must give thanks... I hope it helps.
You will need to adjust the TIMEOUT (in seconds) to the value that you need. Also, adjust the THREADS to the number of threads to the value that you want to run. Populate the @SERVERS variable however you choose. If you need an understanding of why I use the SSH_OPTs like I do, just let me know. Also, these options will work for SCP. Finally, you will need a key for the SSH_KEY constant. Keep in mind the %RESULTS is a simple shared hash, you can populate this as you see fit. However, if you are going to expand the RESULTS beyond the simple example, you will probably need to review shared_clone from the threads::shared module.
If you need any assistance I will help where I can...
#!/usr/bin/perl -l
use strict;
use threads;
use threads::shared;
use Thread::Queue;
use Time::HiRes qw( usleep time );
use IPC::Open3;
use POSIX qw(:errno_h :sys_wait_h);
use File::Basename;
use FindBin qw( $RealBin $RealScript );
use FileHandle;
BEGIN
{
our $SCRIPT_NAME = $RealScript;
our $SCRIPT_DIR = File::Basename::dirname( $RealBin );
# if you have a custom lib, add it.
push ( @INC, ( q{.}, qq{$SCRIPT_DIR/lib} ) );
}
use constant {
RET_SUCCESS => 1,
RET_FAILURE => 0,
EXIT_SUCCESS => 0,
EXIT_FAILURE => 1,
TIMEOUT => 4, ## SET YOUR TIMEOUT in seconds
THREADS => 2, ## SET the number of threads you want to run.
SSH_USER => q{root}, ## Remote user
SSH_KEY => qq{$main::SCRIPT_DIR/keys/YOUR_DSS_KEY}, ## authorize
+d keys
SSH_CMD => q{/usr/bin/ssh},
SSH_OPT => q{-q -o UserKnownHostsFile=/dev/null } .
q{-o StrictHostKeyChecking=no } .
q{-o BatchMode=yes } .
q{-o ConnectTimeout=10 } .
q{-o NoHostAuthenticationForLocalhost=yes } .
q{-o PreferredAuthentications=publickey } .
q{-o ServerAliveInterval=15 } .
q{-o ServerAliveCountMax=4 } .
q{-o TCPKeepAlive=no},
};
my %RESULTS :shared; # Stupid example of collective
# returned information.
my %PROCESS_WATCH :shared; # watch external procs.
# pretend like we have obtained some list of servers.
my @SERVERS = qw( server01.your.domain.com
server02.your.domain.com
);
my $semSTD :shared;
sub tprint {
my $tid = threads->tid;
lock $semSTD;
print STDOUT q{[} . timestamp() . q{][} .
$tid . q{]: }, @_;
return RET_SUCCESS;
}
sub timestamp {
return localtime time;
} ## end timestamp.
my $die_early :shared = 0;
$SIG{ INT } = sub {
tprint q{Early termination requested};
$die_early = 1;
};
sub check_process_signal {
my $sig = shift;
if ( WIFEXITED($sig) ) {
tprint q{process normal exit};
return RET_SUCCESS;
} elsif ( WIFSIGNALED($sig) ) {
tprint q{process terminated because of signal};
return RET_FAILURE;
} elsif ( WIFSTOPPED($sig) ) {
tprint q{process is stopped};
return RET_FAILURE;
}
return RET_SUCCESS;
}
sub add_to_process_watch {
my $pid = shift;
lock %PROCESS_WATCH;
$PROCESS_WATCH{$pid} = time;
return RET_SUCCESS;
}
sub remove_from_process_watch {
my $pid = shift;
lock %PROCESS_WATCH;
if ( defined $PROCESS_WATCH{$pid} ) { delete $PROCESS_WATCH{$pid}; }
return RET_SUCCESS;
}
sub set_results {
lock %RESULTS;
($_[0]) ? $RESULTS{'success'}++ : $RESULTS{'failure'}++;
return RET_SUCCESS
}
sub is_pid_alive {
my $pid = shift;
my $status = RET_SUCCESS;
if ( kill(0, $pid) ) { ## Still alive.
$status = RET_SUCCESS;
} elsif ( $! == EPERM ) { ## Changed UID.
$status = RET_SUCCESS;
} elsif ( $! == ESRCH ) { ## Died or zombied.
$status = RET_FAILURE;
} else { ## Could not locate.
$status = RET_FAILURE;
}
return $status;
} ## end is_pid_alive.
sub run_command {
my $o = {
'debug' => q{},
'host' => q{},
'ssh_user' => q{},
'ssh_key' => q{},
'cmd' => q{},
'opt' => q{},
@_,
};
my $cmd = q{};
if ( (defined $o->{'host'}) and
($o->{'host'} ne q{}) ) {
# Remote command.
my $ssh_user = ( $o->{'ssh_user'} ne q{} )
? $o->{'ssh_user'}
: SSH_USER;
my $ssh_key = ( $o->{'ssh_key'} ne q{} )
? $o->{'ssh_key'}
: SSH_KEY;
$cmd = SSH_CMD . qq{ -i $ssh_key } .
SSH_OPT .
qq{ $ssh_user\@$o->{'host'} '$o->{'cmd'} $o->{'opt'}'};
} else {
# Local command.
$cmd = qq{$o->{'cmd'} $o->{'opt'}}
}
my $hdl = {
'stdin' => FileHandle->new,
'stdout' => FileHandle->new,
'stderr' => FileHandle->new,
};
my $pid = eval { open3( $hdl->{'stdin'},
$hdl->{'stdout'},
$hdl->{'stderr'}, qq{$cmd} ) or die $!; };
add_to_process_watch( $pid );
tprint qq{waiting for external process: $pid};
waitpid( $pid, 0 );
my $exit_status = check_process_signal($?);
my $h_ret = {
'stdout' => [$hdl->{'stdout'}->getlines()],
'stderr' => [$hdl->{'stderr'}->getlines()],
'status' => $exit_status
};
return $h_ret;
}
sub get_disk_information {
my $o = {
'debug' => 0,
'host' => q{},
'df_opt' => q{-Pk},
@_
};
my $ref_cmd = run_command( 'debug' => $o->{'debug'},
'cmd' => q{/bin/df},
'opt' => $o->{'df_opt'},
'host' => $o->{'host'} );
# one could parse the data as needed
# but I will just return the info...
return $ref_cmd;
}
sub get_uname_information {
my $o = {
'debug' => 0,
'host' => q{},
'uname_opt' => q{},
@_
};
my $ref_cmd = run_command( 'debug' => $o->{'debug'},
'cmd' => q{/bin/uname},
'opt' => $o->{'uname_opt'},
'host' => $o->{'host'} );
# one could parse the data as needed
# but I will just return the info...
return $ref_cmd;
}
sub worker {
my( $Q ) = @_;
tprint q{worker started};
while( !$die_early and defined( my $job = $Q->dequeue ) ) {
tprint qq{processing job: $job};
my $ref_di = get_disk_information( 'debug' => 0,
'host' => $job,
'df_opt' => q{-Pk /opt} );
# process the return
( $ref_di->{'status'} )
? tprint q{this thread might continue}
: tprint q{this thread might move on to the next job};
# one might just print the data
tprint qq{stdout: $job\n}, @{$ref_di->{'stdout'}}
if ( $#{$ref_di->{'stdout'}} > 0 );
tprint qq{stderr: $job\n}, @{$ref_di->{'stderr'}}
if ( $#{$ref_di->{'stderr'}} > 0 );
# one might just want to know overall status.
set_results( $ref_di->{'status'} );
# OBVIOUSLY I could call get_uname_information
# and collect and report on that information as
# well. Just an example.
} ## end while
tprint q{Worker ending};
return RET_SUCCESS;
} ## end worker.
my $semPW :shared;
sub process_watcher {
lock $semPW;
while ( !$die_early ) {
usleep( 250_000 );
{
lock %PROCESS_WATCH;
foreach ( keys %PROCESS_WATCH ) {
unless ( ( defined $PROCESS_WATCH{$_} ) and
( is_pid_alive( $_ ) ) and
( ( time - $PROCESS_WATCH{$_} ) > TIMEOUT ) ) { next;
+ }
tprint qq{process $_ exceeded timeout } .
( time - $PROCESS_WATCH{$_} ) ;
kill( 9, $_ );
$PROCESS_WATCH{$_} = undef;
} ## end foreach.
} ## end lock.
} ## end while.
tprint q{process_watcher is terminating};
return RET_SUCCESS;
}
sub main {
%RESULTS = ( 'success' => 0, 'failure' => 0 );
my $Q = new Thread::Queue;
$Q->enqueue( @SERVERS );
$Q->enqueue( (undef) x THREADS );
tprint q{queue populated};
my $thr_pw = threads->create( \&process_watcher )->detach;
my @threads = map threads->new( \&worker, $Q ), 1 .. THREADS;
tprint q{workers started; waiting...};
$_->join for @threads;
print STDOUT q{*} x60;
print STDOUT q{ Success: }, $RESULTS{'success'};
print STDOUT q{ Failure: }, $RESULTS{'failure'};
print STDOUT q{*} x60;
print STDOUT q{Program complete};
return RET_SUCCESS;
}
main();
exit EXIT_SUCCESS;
__END__
| [reply]
[d/l] |