Hello, I have created a script that SSHs to multiple nodes (~150) and issues commands and then process the output. I have used this script successfully in the past but the cli commands produced a short output. Now the cli command I am using produces ~70MB output. Initially I tested using 3 nodes and it was successful. However, when I run it on all nodes, the output (as seen from logs) is cut short in most nodes. You can see that I have set the ssh output timeout at 240s but it doesnt take that long in reality. I am not sure in what issue I am running into but seems to be performance related since it doesn't happen when the number of nodes (and children processes) is low. 
foreach my $node (@nodes) {

    #sleep(1);
    my $pid = $pm->start and next;
    
    
    my $nodeip = $Configuration::node_list_nmnet{$node};
    $datestamp = strftime("%Y%m%d%H%M", localtime);
    
    
    my $ssh = Net::SSH::Expect->new (
    host => $nodeip,
    password=> "*****",
    user => "******",
    timeout => 5,
    raw_pty => 1,
    log_stdout => 0,
    exp_debug => 0,
    log_file => "/home/logs/$node/diam.$node.$datestamp.log"
    );
    
    my $login_output = $ssh->login();
    
    $ssh->waitfor("#", 30);        
    $ssh->send("show diameter peers full debug");
    $ssh->waitfor("#", 240);
    my $output;
    
    $output = $ssh->before;
        if ($output =~ /Peers in CLOSED state\S*\s*(\d*)/ ){
            $peersvalue = $1;
                if ($peersvalue > 0){
                    print "$node\n";
                    push @emaillog, "$node\n\n";
                    
                    $sendemail = 1;
                    
                }
        }
$datestamp = strftime("%Y%m%d%H%M", localtime);

$pm->finish(0);
    
}
print "Waiting for Children...\n";
$pm->wait_all_children;

[download]

In reply to ssh output is partial when using fork manager by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.