Andy16 has asked for the wisdom of the Perl Monks concerning the following question:

Hi monks,

I did a little script to run ssh towards network equipment.
Works well - but now we discovered that sometimes (like 1 out of 5 runs) for a "show running config" some lines are missing...
"show running confog" gives some thousands lines and somewhere in the middle every now and than there is a block of lines missing...

Any idea????

realy every hint is welcome - this is IT and shall be deterministic :->

thanks a lot!!

===== Summary of my perl5 (revision 5 version 10 subversion 0) configuration +: Platform: osname=linux, osvers=2.4.21-297-default, archname=i686-linux-threa +d-multi uname='linux gila 2.4.21-297-default #1 sat jul 23 07:47:39 utc 20 +05 i686 i686 i386 gnulinux ' config_args='-ders -Dcc=gcc -Dusethreads -Duseithreads -Ud_sigsetj +mp -Uinstallusrbinperl -Ulocincpth= -Uloclibpth= -Accflags=-DUSE_SITE +CUSTOMIZE -Duselargefiles -Accflags=-DPRIVLIB_LAST_IN_INC -Dprefix=/N +NM/perl -Dprivlib=/NNM/perl/lib -Darchlib=/NNM/perl/lib -Dsiteprefix= +/NNM/perl/site -Dsitelib=/NNM/perl/site/lib -Dsitearch=/NNM/perl/site +/lib -Dsed=/bin/sed -Duseshrplib -Dcf_by=ActiveState -Dcf_email=suppo +rt@ActiveState.com' ===== package Net::OpenSSH; our $VERSION = '0.60'; =====
use strict; use warnings; use Data::Dumper; use Net::OpenSSH; use Sys::SigAction qw( set_sig_handler sig_alarm timeout_call); use File::Path 'rmtree'; #$Net::OpenSSH::debug = -1; my $sshuser="user"; my $sshpwd="pwd"; my $sshtimeout = 90; # sec for timeout my $workertimeout = 120; my @rawout=(); my $device = $ARGV[0]; my $remoteCmd = $ARGV[1]; my $error; my $ssh; my $path = "./Y.ssh.$$"; my @sshcmds= split(';', $remoteCmd); foreach my $sshcmd (@sshcmds) { rmtree([ $path ]); $ssh = Net::OpenSSH->new( $device, user => $sshuser, passwd => $sshpwd, ctl_dir => $path, master_opts => [-o => "StrictHostKeyChecking=no", -o => "UserK +nownHostsFile=/dev/null"], timeout => $sshtimeout, kill_ssh_on_timeout => 1, strict_mode => 0, master_stderr_discard => 1 # to get rid of login screen ); if ($ssh->error) { $error = $ssh->error; chomp($error); print STDOUT "ERROR-1: " . $error . "\n"; rmtree([ $path ]); exit 1; }; if (timeout_call($workertimeout, sub{ my @cmdout = $ssh->capture({timeout => $sshtimeout, stdin_data + => "\n"}, $sshcmd."\n"); if ($ssh->error) { $error = $ssh->error; chomp($error); print STDOUT "ERROR-2: " . $error . "\n"; rmtree([ $path ]); exit 2; }; push @rawout, @cmdout; } )) { # time out action print STDOUT "ERROR-3: SSH cancled by thread after $workertime +out secs\n"; rmtree([ $path ]); exit 3; }; if ($ssh->error) { $error = $ssh->error; chomp($error); print STDOUT "ERROR-4: SSH: " . $error ."\n"; rmtree([ $path ]); exit 4; }; $ssh->master_exited; } rmtree([ $path ]); foreach (@rawout) { print STDOUT $_; } exit 0;

Replies are listed 'Best First'.
Re: Net::OpenSSH loosing lines ins reply
by salva (Canon) on Jun 04, 2014 at 08:04 UTC
    In order to determine if the problem lays on Net::OpenSSH or on the SSH layer, instead of capturing the output with capture, redirect it to some file:
    $index++; $ssh->system({ timeout => $sshtimeout, stdin_data => "\n", stdout_file => "/tmp/stdout_${index}.txt" }, $sshcmd."\n");

    Then look into those files, if there is still some missing data, it would mean that the problem is on the SSH layer, probably on the SSH server. Otherwise, the more likely cause of the problem would be some bug on Net::OpenSSH.

    It is not uncommon for network equipment to have defective SSH implementations. When that happens, if you are lucky, a firmware update may solve it. Otherwise, you may need to employ some ugly workaround like activating pagination and using Expect to request pages.

      Hi Salva,

      thanks for picking that up!

      I changed the code:
      if (timeout_call($workertimeout, sub{ $ssh->system({ timeout => $sshtimeout, stdin_data => "\n", stdout_file => "/tmp/ssh_del_$$.txt" }, $sshcmd."\n"); open FILE, "<", "/tmp/ssh_del_$$.txt"; my @cmdout = <FILE>; close(FILE); # my @cmdout = $ssh->capture({timeout => $sshtimeout, stdin_da +ta => "\n"}, $sshcmd."\n"); # if ($ssh->error) # { # $error = $ssh->error; # chomp($error); # print STDOUT "ERROR-2: " . $error . "\n"; # rmtree([ $path ]); # exit 2; # }; push @rawout, @cmdout; }


      and tested again.

      Funny:
      I just count the number of lines in the output - and it still differs!
      BUT: the files in /tmp are all the same size!!!

      strange, or?

        That probably means that there are some bug on the server SSH implementation (the other possibility being a bug on the OpenSSH ssh client which is very stable and mature and unlikely to have bugs like that).
        ... and even more fun....

        changed the printing part of the script:

        my $tmpcnt=0; open TWO, ">", "./two.out"; foreach (@rawout) { print "$tmpcnt: ". $_; print TWO "$tmpcnt: ". $_; $tmpcnt++; }


        after running I "tee" stdout to a file and "wc -l" after "tee".
        Sometimes "wc -l" shows a reduced number.
        Then, the tee'ed file also misses a block of lines.
        BUT two.out contains all of them.

        argh!