We have a master-slave setup of Jenkins and the Perl script runs from Jenkins slave. I have written a Perl script which copies deliverable on machine A (and also takes a backup of the same on another machine B) and then invokes a shell script which is already present on machine A. This shell script deploys the deliverable (generally a war file) on machine A. Though this whole task of copying and invoking the shell script could have been written in bash quiet easily, I thought of using Perl just because I had not written Perl program for quite some time. 
use strict;
use warnings;
use Cwd;
use File::Copy;
use Getopt::Long;
use File::Basename;
use Net::OpenSSH;

my ($conf_file, $environment, $docopy, $copy, $doexec, $exec, $job, $d
+est_file, $user, $host, $IP, $TARGET_SERVER, $JENKINS_JOB, $wrapper, 
+$src_file, $src_path, $src_dist_path, $src_full_path, $archive_path, 
+$archive_host, $archive_user, $archive_user_id_file, $id_file, $ssh, 
+$ssh_archive, @array, $line);

usage() if ( @ARGV < 4 or
            !GetOptions(
                    'docopy=s' => \$copy,
                    'doexec=s' => \$exec,
            )
        );

printf "Copy arg: $copy\n";
printf "Exec arg: $exec\n";

sub usage {
    printf "\nEither one of the following arguments are mandatory: doc
+opy / doexec command\n";
    printf "Exiting job...\n\n";
    exit 1;
}


if($ENV{'Environment'} eq "Generic") {
    $IP = $ENV{'TARGET_SERVER'};
    $archive_path = "/home/ec2-user/ite_builds_archive";

    printf "Defined copy\n" if defined $copy;
    printf "Defined exec\n" if defined $exec;

    init();
}


sub init {
    $JENKINS_JOB = $ENV{'JOB_NAME'};

    $conf_file = "/home/ec2-user/SCM/conf/deploy_build.conf";
    open (FH, "<", $conf_file) or die "Cannot open < $conf_file: $!";

    while (<FH>) {
        if ( $_ =~ /\b$JENKINS_JOB\b/ ) {
            push @array, $_;
        } else {
            next;
        }
    }

    printf "\n\nJobs to work on are:\n";
    printf @array;
    printf "\n";

    $archive_host = "10.123.123.100";
    $archive_user = "ec2-user";
    $archive_user_id_file = "/home/ec2-user/.ssh/sandy";

    $ssh_archive = Net::OpenSSH->new($archive_host, key_path => $archi
+ve_user_id_file, user => $archive_user);
    $ssh_archive->error and die "Couldn't establish SSH connection: ".
+ $ssh->error;

    foreach $line (@array) {
        ($job, $src_path, $dest_file, $user, $wrapper) = split(':', $l
+ine);

        if ($dest_file eq "") {
            ($src_file, $src_dist_path) = fileparse($src_path);
            $dest_file = $src_file;
        }    

        printf "Job:         $job\n";
        printf "User:        $user\n";
        printf "Target Machine:    $IP\n";
        printf "Conf File:    $conf_file\n";
        printf "Source Path:     $src_path\n";
        printf "Dest File:    $dest_file\n";
        printf "Wrapper Script:    $wrapper\n";

        $id_file = "/home/ec2-user/.ssh/sandy";

        $ssh = Net::OpenSSH->new($IP, key_path => $id_file, user => $u
+ser);
        $ssh->error and die "Couldn't establish SSH connection: ". $ss
+h->error;

        if (defined $copy) {
            printf "\n";
            printf "Initiating subroutine to copy distributable on rem
+ote machine...\n";
            &copy_distributable;
        } 

        if (defined $exec) {
            printf "\n";
            if (length $wrapper) {        
                printf "Initiating subroutine for executing wrapper on
+ remote machine...\n";
                &exec_wrapper;
            } else {
                printf "*** No wrapper specified ****\n";
            }
        }
    }
}


sub copy_distributable {
    $src_full_path = "$ENV{WORKSPACE}/$src_path";
    
    if ( -f $src_full_path ) {
        if ($dest_file ne "") {
            printf "Distributable:    $dest_file\n";
        }

        printf "User:        $user\n";
        printf "Source Path:    $src_full_path\n";
        printf "Target Machine:    $IP\n\n";

        $ssh_archive->scp_put("$src_full_path", "$archive_path/$dest_f
+ile");    
        $ssh_archive->error and die "ERROR: Couldn't archive deliverab
+le on Jenkins master\n". $ssh_archive->error;

        $ssh->scp_put("$src_full_path", "/home/$user/$dest_file");
        $ssh->error and die "ERROR: SCP to target machine failed!\n". 
+$ssh->error;

        printf "Deliverable copied on deployment machine. Now moving o
+n to next task of archiving the deliverable...\n\n";

        printf "mv $archive_path/$dest_file $archive_path/latest/\n\n"
+;
        my ($stdout) = $ssh_archive->capture2("mv $archive_path/$dest_
+file $archive_path/latest/");
        $ssh_archive->error and die "Remote command to move file to $a
+rchive_path/latest/ directory failed: " . $ssh_archive->error;

        printf "Output: $stdout\n" if $stdout;
        printf "Deliverable archived on Jenkins master\n";
    } else {
        printf "Deliverable not found at $src_full_path\n";
        exit 1;
    }
}


sub exec_wrapper {
    printf "Wrapper to be executed: $wrapper\n";
    printf "Target Machine: $IP\n";
    printf "User: $user\n\n";

    my ($stdout) = $ssh->capture2("~/release/$wrapper");
    $ssh->error and die "Remote command failed to execute wrapper ~/re
+lease/$wrapper: " . $ssh->error;

    printf "Output: $stdout\n" if $stdout;
}

[download]

Details of Jenkins slave box where the build workspace is:

[ec2-user@jenkins_slave2 ~]$ uname -a
Linux jenkins_slave 3.10.35-43.137.amzn1.x86_64 #1 SMP Wed Apr 2 09:36
+:59 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

[ec2-user@jenkins_slave2 ~]$ free -m
              total      used       free       shared   buffers       
+  cached
Mem:          7228       5688       1539          0         72        
+   3824
-/+ buffers/cache:     1791       5436
Swap:        16382       163       16219

[download]

Also, there is sufficient memory on the box where the deliverable is copied and finally deployed. Disk space is also not an issue on either of the machines. Both are Linux instances running on AWS.

Now the issue is that though the program runs fine, many a times, it stucks when the deployment script (shell script) is called. To debug, i ran the Perl script in Jenkins using Devel::Trace. When the build got stuck, I aborted the build. As soon as I aborted the build, I got heap space error so I added Xms & Xmx parameters to set initial and maximum Java heap size respectively. After confirming that the settings are applied, I ran the builds again. Few builds passed and few again got stuck. I don’t think increasing heap size further makes any sense. A shell script was written to do the same task and it does not require any such extra memory to run. Now I am not sure where to look for clues.

Any help will really be appreciated.

UPDATE: The only thing that I have changed in my post is the script. I made small changes based on the feedback I received on this post on SO. Now since i changed my module to Net::OpenSSH (as suggested by salva), I have observed that the script hangs rarely although it's still not as fast as its bash counterpart. My question is: are bash scripts faster when it comes to execution than its perl counterpart? Another query: In some cases, I also notice that a bash script prints messages that my perl script doesn’t. For instance, I had an issue where deployment was not happening on machine A. On investigation, I came to know that Java was not deployed. This I came to know when I logged into machine A and ran the bash script that my perl script was calling. I thought of triggering the same bash script using another bash script remotely from the same machine where my perl script was. I did see the Java missing message. Why is it so? Why is my perl script unable to capture the error message despite using ‘$ssh->error’ in my script? 
[ec2-user@slave work]$ cat test_abc.sh
#/bin/bash
ssh -i /home/ec2-user/.ssh/sandy abc@some_server './deploy.sh'


[ec2-user@slave work]$ ./deploy.sh
Starting with abc deployment
Doing deployment of abc
Stopping abc server /usr/local/abc-tomcat
/usr/local/abc-tomcat/bin/catalina.sh: line 437: /usr/lib/jvm/jdk1.7.0
+_51/bin/java: No such file or directory
Tomcat already stopped for server abc-tomcat no need to forcefull shut
+down
Clearing ROOT directory
Placing the new content in web server
Archive:  /home/abc/abc.war
creating: /usr/local/abc-tomcat/webapps/ROOT/META-INF/
...
...
...

[download]
My perl script doesn’t print the line /usr/local/abc-tomcat/bin/catalina.sh: line 437: /usr/lib/jvm/jdk1.7.0_51/bin/java: No such file or directory

In reply to Perl (Net::SSH::Perl) script sometimes hangs while running remote command but shell script works by Technext

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.