backticks fail to capture stderr even with explicit redirection

keithhanlan has asked for the wisdom of the Perl Monks concerning the following question:

As soon as I posted this question, found the following in man perlop:

A string which is (possibly) interpolated and then executed as a system command with /bin/sh or its equivalent. Shell wildcards, pipes, and redirections will be honored. The collected standard output of the command is returned; standard error is unaffected.

Given the bold statement, I call "bad design" on the last phrase. This has been very frustrating. Perl's philosophy of sometimes guessing what I mean rather than taking what I say at face value has now officially really ticked me off. At the very least, this construct should produce a warning message!

...Original post follows...

Can somebody suggest why backticks aren't able to capture stderr in our environment?

I generally use open() with pipes to capture the output of commands but a colleague was using back-ticks and encountered behaviour that runs counter to sample code I've seen all over the place, namely:

$alloutput=`cmd 2>&1`; # This should capture both stdout and stderr.
[download]

However, it doesn't seem to work in our environment for some reason and I would like to understand why.

Here's a simple test:

% cat asdf
#!/bin/sh
echo "output to stdout"
echo "output to stderr" >&2
[download]

Now, witness the following:

% perl -e '$x=`./asdf 2>&1`;print "Captured $x"'
Captured output to stdout
output to stderr
[download]

Notice how the stderr output escapes the capture. This is contrary to examples that I've seen, for example, here: How can I capture STDERR from an external command? which explicitly states:

    $output = `cmd 2>&1`;                   # either with backticks
    $pid = open(PH, "cmd 2>&1 |");          # or with an open pipe
    while (<PH>) { }                        #    plus a read
[download]

Note that if I use the latter construct, it does work:

% perl -e 'open(F,"./asdf 2>&1|");while($x=<F>){print "Captured $x";}'
Captured output to stdout
Captured output to stderr
[download]

In our environment, this is reproducible on both Linux & Solaris using perl 5.5.3, 5.8.5, and 5.12.2.

Any help or hints would be appreciated.

Thank you,
Keith

Comment on backticks fail to capture stderr even with explicit redirection Select or Download Code

Replies are listed 'Best First'.
Re: backticks fail to capture stderr even with explicit redirection by Tanktalus (Canon) on Jul 04, 2011 at 16:55 UTC
This is why I use begin AND end delimiters on my debug output. Instead of `print "Captured $x"`, try print "Captured >>$x<<". <c>$ perl -le '$x=`./asdf 2>&1`;print ">>$x<<"' >>output to stdout output to stderr << [download] Both are caught here, with all the perls I have available to me. (Normally I use `[$x]` - don't know why I didn't do that here - I guess this just stands out more.)	[reply] [d/l] [select]
Re^2: backticks fail to capture stderr even with explicit redirection by keithhanlan (Initiate) on Jul 06, 2011 at 19:25 UTC
Good point on the delimiters. My test program was flawed. In the environment where we are testing, backticks do not capture the stderr even when I explicitly redirect it to stdout. I'll keep digging. Somebody further down the respondent chain has pointed out an interesting bit about $Config. Thank you, Keith	[reply]
Re^3: backticks fail to capture stderr even with explicit redirection by Tanktalus (Canon) on Jul 06, 2011 at 22:22 UTC
I highly doubt that. Very highly doubt it. Try using a data-dumping module to spit out exactly what you have. For example: $ perl -MJSON -le '$output=`./asdf`; print encode_json([$output])' output to stderr ["output to stdout\n"] $ perl -MJSON -le '$output=`./asdf 2>&1`; print encode_json([$output]) +' ["output to stdout\noutput to stderr\n"] [download] or, if you don't have JSON: $ perl -MData::Dumper -le '$output=`./asdf`;print Dumper $output' output to stderr $VAR1 = 'output to stdout '; $ perl -MData::Dumper -le '$output=`./asdf 2>&1`;print Dumper $output' $VAR1 = 'output to stdout output to stderr '; [download] You'll notice that if you omit the 2>&1, the stderr part shows up outside of the variable being dumped. What's more, it shows up first, which your original code shows is not what is happening for you. The only way that the stderr shows up after the stdout, given your asdf program, is if the stderr is successfully caught. I will now ask you to take back you "bad design" call in your updated original node. It is based on your flawed testing, which your own node shows to be flawed. If you want to understand why the order of the output would be different if you weren't catching stderr, I would suggest a new root-level node for that. Not because we can't answer it here, but because I think you should go back to your test code and play with it for some time to see if you can figure it out yourself (with the help of google, that's fine) before asking.	[reply] [d/l] [select]
Re: backticks fail to capture stderr even with explicit redirection by ikegami (Patriarch) on Jul 04, 2011 at 19:52 UTC
Your conclusions are wrong. STDERR is being captured and stored in `$x`. $ cat asdf #!/bin/sh echo "line 1" echo "line 2" echo "line 3" >&2 echo "line 4" >&2 $ perl -le'$x=`./asdf 2>&1`; print "<<Captured $x>>"' <<Captured line 1 line 2 line 3 line 4 >> [download] Using backticks in list context will split the lines if that's what you want. $ perl -e'print "Captured $_" for `./asdf 2>&1`;' Captured line 1 Captured line 2 Captured line 3 Captured line 4 [download]	[reply] [d/l] [select]
Re: backticks fail to capture stderr even with explicit redirection by hossman (Prior) on Jul 04, 2011 at 18:19 UTC
A string which is (possibly) interpolated and then executed as a system command with /bin/sh or its equivalent. Shell wildcards, pipes, and redirections will be honored. The collected standard output of the command is returned; standard error is unaffected. Given the bold statement, I call "bad design" on the last phrase. This has been very frustrating. Perl's philosophy of sometimes guessing what I mean rather than taking what I say at face value has now officially really ticked me off. At the very least, this construct should produce a warning message! The documentation is correct. The backticks execute /bin/sh and capture the STDOUT of /bin/sh. the string you specify, including the redirection, is processed by /bin/sh -- perl (and hte backticks) don't know or care that you have asked /bin/sh to execut "cmd" and redirect cmd's STDERR to cmd's STDOUT (which then becomes the STDOUT of /bin/sh). All perl does is is take /bin/sh's STDOUT and give that to you. See the previous comments explaining why it doesn't seem like it's working because of your flawed test case The example you said does work would also appear to fail, if you had written it to work the same way (ie: print in single scalar variable in which you have slurped all of the data from F)	[reply]
Re^2: backticks fail to capture stderr even with explicit redirection by Anonymous Monk on Jul 04, 2011 at 19:27 UTC
The backticks (and system, open, etc ...) consult $Config::Config{sh} for which shell to use `$ perl -V:sh sh='cmd /x /c';` [download]	[reply] [d/l]
Re^2: backticks fail to capture stderr even with explicit redirection by keithhanlan (Initiate) on Jul 06, 2011 at 19:33 UTC
The example you said does work would also appear to fail, if you had written it to work the same way (ie: print in single scalar variable in which you have slurped all of the data from F) Doh! good catch. Thanks for pointing that out.	[reply]
Re: backticks fail to capture stderr even with explicit redirection by JavaFan (Canon) on Jul 04, 2011 at 18:02 UTC
Can somebody suggest why backticks aren't able to capture stderr in our environment? That's because the shell does it that way. Considering that it's rather easy to merge streams, and rather hard to separate merged streams, I'm rather glad that backticks don't merge STDOUT and STDERR.	[reply]
Re^2: backticks fail to capture stderr even with explicit redirection by keithhanlan (Initiate) on Jul 06, 2011 at 19:29 UTC
Perhaps I didn't phrase my question properly. I should have written Can somebody suggest why subshells launched by backticks don't let me redirect stderr? You'll notice that I am trying to explicitly merge stdout & stderr. The problem is that backticks ignore my explicit direction. Refer to the updated comments at the top of my original post for more information. Thank you, Keith	[reply]
Re^3: backticks fail to capture stderr even with explicit redirection by JavaFan (Canon) on Jul 06, 2011 at 21:26 UTC
You'll notice that I am trying to explicitly merge stdout & stderr. The problem is that backticks ignore my explicit direction. Uhm, you are capturing both STDOUT and STDERR. Other than that your test program is written in such a way that it neither proves, nor disproves that you are capturing both, what makes you think it doesn't? Hint, change your `print "Captured $x"` in `print "Captured --->$x<---\n"` and it should be obvious.	[reply] [d/l] [select]
Re: backticks fail to capture stderr even with explicit redirection by reisub (Initiate) on Jan 19, 2012 at 03:57 UTC
Hey, instead of :/$ perl -e '$alloutput=`cmd 2>&1`; print $alloutput;' :/$ [download] try that :/$ perl -e '$alloutput=`cmd 2>&1;`; print $alloutput;' sh: cmd: command not found :/$ [download] Regards.	[reply] [d/l] [select]