Technext has asked for the wisdom of the Perl Monks concerning the following question:

We have a master-slave setup of Jenkins and the Perl script runs from Jenkins slave. I have written a Perl script which copies deliverable on machine A (and also takes a backup of the same on another machine B) and then invokes a shell script which is already present on machine A. This shell script deploys the deliverable (generally a war file) on machine A. Though this whole task of copying and invoking the shell script could have been written in bash quiet easily, I thought of using Perl just because I had not written Perl program for quite some time. 
use strict;
use warnings;
use Cwd;
use File::Copy;
use Getopt::Long;
use File::Basename;
use Net::OpenSSH;

my ($conf_file, $environment, $docopy, $copy, $doexec, $exec, $job, $d
+est_file, $user, $host, $IP, $TARGET_SERVER, $JENKINS_JOB, $wrapper, 
+$src_file, $src_path, $src_dist_path, $src_full_path, $archive_path, 
+$archive_host, $archive_user, $archive_user_id_file, $id_file, $ssh, 
+$ssh_archive, @array, $line);

usage() if ( @ARGV < 4 or
            !GetOptions(
                    'docopy=s' => \$copy,
                    'doexec=s' => \$exec,
            )
        );

printf "Copy arg: $copy\n";
printf "Exec arg: $exec\n";

sub usage {
    printf "\nEither one of the following arguments are mandatory: doc
+opy / doexec command\n";
    printf "Exiting job...\n\n";
    exit 1;
}


if($ENV{'Environment'} eq "Generic") {
    $IP = $ENV{'TARGET_SERVER'};
    $archive_path = "/home/ec2-user/ite_builds_archive";

    printf "Defined copy\n" if defined $copy;
    printf "Defined exec\n" if defined $exec;

    init();
}


sub init {
    $JENKINS_JOB = $ENV{'JOB_NAME'};

    $conf_file = "/home/ec2-user/SCM/conf/deploy_build.conf";
    open (FH, "<", $conf_file) or die "Cannot open < $conf_file: $!";

    while (<FH>) {
        if ( $_ =~ /\b$JENKINS_JOB\b/ ) {
            push @array, $_;
        } else {
            next;
        }
    }

    printf "\n\nJobs to work on are:\n";
    printf @array;
    printf "\n";

    $archive_host = "10.123.123.100";
    $archive_user = "ec2-user";
    $archive_user_id_file = "/home/ec2-user/.ssh/sandy";

    $ssh_archive = Net::OpenSSH->new($archive_host, key_path => $archi
+ve_user_id_file, user => $archive_user);
    $ssh_archive->error and die "Couldn't establish SSH connection: ".
+ $ssh->error;

    foreach $line (@array) {
        ($job, $src_path, $dest_file, $user, $wrapper) = split(':', $l
+ine);

        if ($dest_file eq "") {
            ($src_file, $src_dist_path) = fileparse($src_path);
            $dest_file = $src_file;
        }    

        printf "Job:         $job\n";
        printf "User:        $user\n";
        printf "Target Machine:    $IP\n";
        printf "Conf File:    $conf_file\n";
        printf "Source Path:     $src_path\n";
        printf "Dest File:    $dest_file\n";
        printf "Wrapper Script:    $wrapper\n";

        $id_file = "/home/ec2-user/.ssh/sandy";

        $ssh = Net::OpenSSH->new($IP, key_path => $id_file, user => $u
+ser);
        $ssh->error and die "Couldn't establish SSH connection: ". $ss
+h->error;

        if (defined $copy) {
            printf "\n";
            printf "Initiating subroutine to copy distributable on rem
+ote machine...\n";
            &copy_distributable;
        } 

        if (defined $exec) {
            printf "\n";
            if (length $wrapper) {        
                printf "Initiating subroutine for executing wrapper on
+ remote machine...\n";
                &exec_wrapper;
            } else {
                printf "*** No wrapper specified ****\n";
            }
        }
    }
}


sub copy_distributable {
    $src_full_path = "$ENV{WORKSPACE}/$src_path";
    
    if ( -f $src_full_path ) {
        if ($dest_file ne "") {
            printf "Distributable:    $dest_file\n";
        }

        printf "User:        $user\n";
        printf "Source Path:    $src_full_path\n";
        printf "Target Machine:    $IP\n\n";

        $ssh_archive->scp_put("$src_full_path", "$archive_path/$dest_f
+ile");    
        $ssh_archive->error and die "ERROR: Couldn't archive deliverab
+le on Jenkins master\n". $ssh_archive->error;

        $ssh->scp_put("$src_full_path", "/home/$user/$dest_file");
        $ssh->error and die "ERROR: SCP to target machine failed!\n". 
+$ssh->error;

        printf "Deliverable copied on deployment machine. Now moving o
+n to next task of archiving the deliverable...\n\n";

        printf "mv $archive_path/$dest_file $archive_path/latest/\n\n"
+;
        my ($stdout) = $ssh_archive->capture2("mv $archive_path/$dest_
+file $archive_path/latest/");
        $ssh_archive->error and die "Remote command to move file to $a
+rchive_path/latest/ directory failed: " . $ssh_archive->error;

        printf "Output: $stdout\n" if $stdout;
        printf "Deliverable archived on Jenkins master\n";
    } else {
        printf "Deliverable not found at $src_full_path\n";
        exit 1;
    }
}


sub exec_wrapper {
    printf "Wrapper to be executed: $wrapper\n";
    printf "Target Machine: $IP\n";
    printf "User: $user\n\n";

    my ($stdout) = $ssh->capture2("~/release/$wrapper");
    $ssh->error and die "Remote command failed to execute wrapper ~/re
+lease/$wrapper: " . $ssh->error;

    printf "Output: $stdout\n" if $stdout;
}

[download]

Details of Jenkins slave box where the build workspace is:

[ec2-user@jenkins_slave2 ~]$ uname -a
Linux jenkins_slave 3.10.35-43.137.amzn1.x86_64 #1 SMP Wed Apr 2 09:36
+:59 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

[ec2-user@jenkins_slave2 ~]$ free -m
              total      used       free       shared   buffers       
+  cached
Mem:          7228       5688       1539          0         72        
+   3824
-/+ buffers/cache:     1791       5436
Swap:        16382       163       16219

[download]

Also, there is sufficient memory on the box where the deliverable is copied and finally deployed. Disk space is also not an issue on either of the machines. Both are Linux instances running on AWS.

Now the issue is that though the program runs fine, many a times, it stucks when the deployment script (shell script) is called. To debug, i ran the Perl script in Jenkins using Devel::Trace. When the build got stuck, I aborted the build. As soon as I aborted the build, I got heap space error so I added Xms & Xmx parameters to set initial and maximum Java heap size respectively. After confirming that the settings are applied, I ran the builds again. Few builds passed and few again got stuck. I don’t think increasing heap size further makes any sense. A shell script was written to do the same task and it does not require any such extra memory to run. Now I am not sure where to look for clues.

Any help will really be appreciated.

UPDATE: The only thing that I have changed in my post is the script. I made small changes based on the feedback I received on this post on SO. Now since i changed my module to Net::OpenSSH (as suggested by salva), I have observed that the script hangs rarely although it's still not as fast as its bash counterpart. My question is: are bash scripts faster when it comes to execution than its perl counterpart? Another query: In some cases, I also notice that a bash script prints messages that my perl script doesn’t. For instance, I had an issue where deployment was not happening on machine A. On investigation, I came to know that Java was not deployed. This I came to know when I logged into machine A and ran the bash script that my perl script was calling. I thought of triggering the same bash script using another bash script remotely from the same machine where my perl script was. I did see the Java missing message. Why is it so? Why is my perl script unable to capture the error message despite using ‘$ssh->error’ in my script? 
[ec2-user@slave work]$ cat test_abc.sh
#/bin/bash
ssh -i /home/ec2-user/.ssh/sandy abc@some_server './deploy.sh'


[ec2-user@slave work]$ ./deploy.sh
Starting with abc deployment
Doing deployment of abc
Stopping abc server /usr/local/abc-tomcat
/usr/local/abc-tomcat/bin/catalina.sh: line 437: /usr/lib/jvm/jdk1.7.0
+_51/bin/java: No such file or directory
Tomcat already stopped for server abc-tomcat no need to forcefull shut
+down
Clearing ROOT directory
Placing the new content in web server
Archive:  /home/abc/abc.war
creating: /usr/local/abc-tomcat/webapps/ROOT/META-INF/
...
...
...

[download]
My perl script doesn’t print the line /usr/local/abc-tomcat/bin/catalina.sh: line 437: /usr/lib/jvm/jdk1.7.0_51/bin/java: No such file or directory
  • Comment on Perl (Net::SSH::Perl) script sometimes hangs while running remote command but shell script works
  • Select or Download Code

Replies are listed 'Best First'.
Re: Perl (Net::SSH::Perl) script sometimes hangs while running remote command but shell script works
by salva (Canon) on Aug 01, 2014 at 06:59 UTC
    And where does it get stuck?

    Anyway, try replacing Net::SSH::Perl and Net::SCP::Expect usage by Net::OpenSSH.

[reply]
      It gets stuck only when it's invoking a wrapper script present on remote machine. SCP works fine.
[reply]
        Then, connect to the remote server in parallel and inspect what the wrapper script is doing (for instance, using strace or a debugger), it should be waiting for something.
[reply]
[d/l]
Re: Perl (Net::SSH::Perl) script sometimes hangs while running remote command but shell script works
by salva (Canon) on Jun 16, 2015 at 08:55 UTC
    I have observed that the script hangs rarely
    That indicates a bug somewhere. Use strace or some similar tool to discover what every component is doing. Your Java application or the wrapper may be waiting for something.

    You can also enable debugging on Net::OpenSSH to see what is happening at the perl level: 

    $Net::OpenSSH::debug = -1;

    [download]

    are bash scripts faster when it comes to execution than its perl counterpart?

    Perl code is usually faster than shell code. On scripts that invoke external commands, there is usually no difference between the two.

    If your Perl script is slower it probably means that it is doing things in a different way.

    In some cases, I also notice that a bash script prints messages that my perl script doesn’t.

    Net::OpenSSH provides you several methods to redirect or capture the stdio streams of the remote process. You have to pick the ones that you need.

    For instance, I had an issue where deployment was not happening on machine A. On investigation, I came to know that Java was not deployed. This I came to know when I logged into machine A and ran the bash script that my perl script was calling. I thought of triggering the same bash script using another bash script remotely from the same machine where my perl script was. I did see the Java missing message. Why is it so? Why is my perl script unable to capture the error message despite using ‘$ssh->error’ in my script?

    That means that your wrapper script is hiding it. $ssh->error returns an error code when the remote command exits with a non zero value.

[reply]
[d/l]
[select]
      Thanks for the help salva! As far as debugging the script (when it hangs) is concerned, I’ll use strace.

      For the issue where bash was printing error message but my perl script wasn’t, I figured out the mistake I was committing. Instead of

      my ($stdout) = $ssh->capture2("~/release/$wrapper");

      I should have used

      my ($stdout, $errput) = $ssh->capture2("~/release/$wrapper");

      I was only checking stdout earlier :)

[reply]
[d/l]
[select]

        There is one issue that i noticed today. If i use the following code, it dies after printing error message. Obviously, the stdout will not be printed

        die "Error: $errput\n" if $errput;
printf "Output: $stdout\n" if $stdout;

        [download]
        Problem is when i reverse the lines as shown below, it prints the output and then dies but does not print the error message. 
        printf "Output: $stdout\n" if $stdout;
die "Error: $errput\n" if $errput;

        [download]
        How can i make it print the stdout and the error message both?
[reply]
[d/l]
[select]

Back to Seekers of Perl Wisdom