devnull has asked for the wisdom of the Perl Monks concerning the following question:

Hi I just posted a note on "Re^2: best practices for checking system return values for piped commands?", but figured I really needed to post a question, so apologies if this seems like a double post...
I'm struggling with trying to identify a failure in a set of piped commands run on a remote host via ssh. I use perl to build and run the command line as below 
ssh $TARGET_HOST "sudo find $TARGET_DIR -name \'$TARGET_FILE\' |
sudo cpio --create --format ustar" |
bzip2 -9 -c

[download]
Initially I just redirected the output to a file, which is fine if both the $TARGET_DIR and $TARGET_FILE exists. However, if one or both do not exist, then I still get a file created, but it's an empty .tar.bz2 file of a few bytes.
Things I've tried:
With $ret=system($cmd); and testing the the value of $ret:
  • Pre-fixing the cmd with set -o pipefail &&.
  • Adding set -o pipefail to the command run by ssh.
  • Adding set -o pipefail to the command run by sudo.
  • Doing the bzip2 locally.
    With @results = `$cmd`, and testing the value of $?:
  • Adding ; echo \${PIPESTATUS\@}; to the end of $cmd, within the string passed to ssh.
  • Adding ; echo \${PIPESTATUS\@}; to the end of $cmd, after the string passed to ssh.
    Using the @results approach, I only get one entry, so I suspect that the backtick method only sees one stream?

    Any ideas on how I can detect the remote pipeline failure? I'm using perl 5.8.5, bash 3.00.15(1) over RedHat ELAS4.5. I would not be permitted to install the Net::Ssh module.

    • Replies are listed 'Best First'.
    Re: Detecting failed piped command run via ssh
    by cdarke (Prior) on Apr 08, 2009 at 09:18 UTC
      Your Bash syntax for accessing all the elements of the PIPESTATUS array needs looking at. It should be:
      ${PIPESTATUS[@]}

      [download]
      With appropriate backslashing:
      \$\{PIPESTATUS\[\@\]\}

      [download]
      You only get one element because using an array without an index gives the first element (in Bash and ksh).
    		[reply]
    [d/l]
    [select]
        Thanks, but I still get only one entry in @results.
        I figure I should get one entry as the contents of the .tar.bz2, and one as whatever is returned by  ;echo \$\{PIPESTATUS\[\@\]\}. Later code writes $results[0] to a file, which bunzip2 reports as corrupt, hence I figure that the backtick method is unable to separate the output of bzip2 -9 -c from  ;echo \$\{PIPESTATUS\[\@\]\}.
    		[reply]
    [d/l]
    [select]
          When you say there is only one entry in @results, doea that one entry contain output from bzip2, or is it from PIPESTATUS? It could just be that bzip2 is not terminating the record with "\n", in which case inserting a dummy echo after the bzip2 should do the trick.
    		[reply]
    Re: Detecting failed piped command run via ssh
    by Bloodnok (Vicar) on Apr 08, 2009 at 09:27 UTC
      I know it's not a direct solution to your problem, but I recently encountered a different approach to a similar problem whereby the...
      1. pipeline was split into its individual commands
      2. each of the commands is echoed to a temporary sh(1) script on the host
      3. the temporary script is then run on the host using ssh(1) & sh(1)

      e.g. (escaping as required) 

      # ssh $HOST "cat > /tmp/script" << !
rm -f /tmp/files /tmp/contents
sudo find $TARGET_DIR -name \'$TARGET_FILE\' > /tmp/files || exit $?
sudo cat /tmp/files | cpio -oBO /tmp/contents --format ustar" || exit 
+$?
bzip2 -9 -c /tmp/contents || exit $?
exit 0
!
# ssh $HOST "sudo sh /tmp/script"

      [download]
      ... or similar :-D

      A user level that continues to overstate my experience :-))
    		[reply]
    [d/l]
    [select]
        Thanks, I am sure that this approach would work, but I am a little reluctant due to the race conditions, security considerations and extra code. The "ssh/find/cpio/bzip2" approach is simple, and the only drawback is not telling the user that they've failed to get the right dir/file is probably acceptable - it just doesn't give that satisfaction of doing a good job.
    		[reply]

    Back to Seekers of Perl Wisdom