in reply to Re^2: use of Backticks to catch console output
in thread use of Backticks to catch console output

G'day haukex,

"Just a small nitpick: on Windows it is almost impossible to avoid the shell."

I checked out that module's documentation and code not so long; I've just checked again. Both are still very clear that capturex() does not invoke the shell. Here's a selection of extracts (non-exhaustive):

From SYNOPSIS:

# As above, but NEVER invokes the shell. my $output = capturex("some_command", @args);

From source, starting at Line 361 (note the "NO_SHELL"):

# capturex() is just like backticks/qx, but never invokes the shell. sub capturex { ... if (WINDOWS) { return _win32_capture(NO_SHELL, $valid_returns, $command, +@args); }

Are you possibly confusing capturex() with capture()? Same source, starting at Line 220 (note the "USE_SHELL"):

# capture is our way of running a process with backticks/qx semantics sub capture { ... if (WINDOWS) { # USE_SHELL really means "You may use the shell if you nee +d it." return _win32_capture(USE_SHELL, $valid_returns, $command) +; }

If not a case of confusion, do you think the documentation, code, or something else, is wrong?

All links and extracts are from the IPC-System-Simple-1.30 distribution (released "Mar 24, 2020").

— Ken

Replies are listed 'Best First'.
Re^4: use of Backticks to catch console output
by haukex (Archbishop) on Dec 11, 2022 at 14:24 UTC
    From source, ...

    You need to trace the source a little further and look into _win32_capture, where you'll see that Win32::ShellQuote gets called no matter what the value of $use_shell is.

    Both are still very clear that capturex() does not invoke the shell.

    In my experience the documentation of such modules is usually tailored to *NIX. Either that or people eqate using Win32::ShellQuote to "avoiding the shell"*, which isn't quite accurate, and the module does have some edge cases that means it's not the same as execvp on *NIX - like I said, I was being nitpicky :-)

    Have a look at Re^2: Having to manually escape quote character in args to "system"? for more details on calling commands on Windows and why argument quoting is such an issue there.

    * Update: See my clarification further down in the thread.

      No, I'm still not seeing it. I'm not at all familiar with modules in the Win32:: namespace; perhaps you can help me out with this.

      "... you'll see that Win32::ShellQuote gets called ..."

      Win32::ShellQuote is a module and gets loaded; I've checked the code and don't see that operation invoking a shell. Win32::ShellQuote::quote_native() gets conditionally called; again, I've checked the code and don't see that operation invoking a shell either.

      quote_native() appears within _spawn_or_die(). Looking into that routine, I see Win32::Process::Create() is called. There's XS code under the hood (I have pretty much zero XS expertise); perhaps a shell is invoked here.

      The impression I get is that Win32::Process::Create() is doing what it's name suggests, i.e. creates a process. If a *nix system, open(my $pipe, '-|') would be used, which also creates (forks) a process.

      — Ken

        I did a quick search and apparently I've written about calling external commands more safely over a hundred times. Perhaps I'm getting a bit tired of it, which caused me to use inaccurate language - in other words, I wasn't nitpicky enough with myself, sorry about that. Specifically, I think I've gotten used to using the phrase "avoiding the shell" as a blanket term. So here's the full story:

        Calling external commands is something I try to do as little as possible and definitely wouldn't do in a tight loop, which is why I don't care all too much about the overhead of calling an external command, including whether a possible shell in between my script and the external command is having a performance impact. What I do care about a lot is being exact in what ends up in the external command's @ARGV. Aside from the major security implications of shell injections, I also care that if I distribute my script to other machines that may have different environments and different shells, I wouldn't want some strange quoting and interpolation issues to trip up potential users. Or, for example, if I put my script onto a server and the admin decides to change what /bin/sh is, my script may mysteriously stop working, or even worse, appear to keep working while actually producing inaccurate results (see also afoken's The problem of "the" default shell). It's admittedly a little bit paranoid, but OTOH, avoiding such issues doesn't take a lot of work, so why not. And wisdom seekers regularly come here with things like in the root node, a variable interpolated into backticks with no indication whether that variable is user input or not, so I prefer to err on the side of caution and assume that it is, which is why I think you provided the only "correct" answer in this thread by suggesting capturex.

        On *NIX, several Perl functions like system and exec, when called correctly, allow one to use execvp, which allows one to know exactly what the external command will get in its @ARGV. As I learned from the node I linked to (among other places), in Windows, command arguments are passed to commands as one big string, and then it's up to the command to parse that string (see also). Though there exist standardized libraries and functions to do so, there are probably a lot of commands out there that reinvented that wheel and have their own quoting, escaping, etc. schemes. What this means is that on Windows, there appears to be no single, correct way to pass arguments to external commands, regardless of whether a shell is present or not - though AFAICT the shell does introduce additional complications. So really, the issue is not "avoiding the shell", which I incorrectly said, but more like "avoiding quoting and escaping issues". (Update: And on *NIX, the two are eqivalent, another reason why I got lazy with the phrase.)

        I pointed to the use of Win32::ShellQuote to indicate that IPC::System::Simple was using what I still consider a "workaround" to the "single command string" problem. Even if there's no shell in between the script and the external command, with the combination of Win32::ShellQuote having to quote the strings in the first place, and the external command having to parse that, there is still a potential for a mismatch between the two. Even though Win32::ShellQuote may work just fine in almost all cases, I personally still think the 1% (or even 0.1%) of cases where there may be mysterious failures when attempting to call external commands on Windows made it worth it to pick that nit. To quote afoken: "On Windows, you simply can not win the quoting game."

        I should add that, at the moment, I am not aware of any security issues when using Win32::ShellQuote to escape arguments passed to external commands, though of course there may be hidden issues or ones that show up in the future. I do recall seeing some issues where the quoting provided by the module didn't work, though I'm having trouble finding examples at the moment. But in any case, IPC::System::Simple (which is nice because its systemx is a drop-in replacement for system and its capturex is a drop-in replacement for backticks, and both provide nice error handling) and IPC::Run3 (which I like because it has a few more features than the former), both use Win32::ShellQuote on Windows, which is still much better than not using it.

        If a *nix system, open(my $pipe, '-|') would be used, which also creates (forks) a process.

        Just a note here: As I explain in my node on the topic, similar to system, piped opens can also invoke the shell or not, depending on the arguments.

        And just for completeness, here's an example showing that "single command string" on Windows:

        showcmdline.pl:

        use warnings; use strict; use Win32::API; use Data::Dump; my $GetCommandLine = Win32::API->new("kernel32", "GetCommandLine", [], + 'P'); ( my $c = pack("a1024", $GetCommandLine->Call()) ) =~ s/\0*$//; print $c, "\n"; dd $0, @ARGV;

        Which I can call like so:

        use warnings; use strict; use IPC::System::Simple qw/capturex/; print capturex( $^X, 'showcmdline.pl', 'foo\\" bar\\\\', 'open(my $x," +>&STDOUT");' ); __END__ "C:\Strawberry\perl\bin\perl.exe" "showcmdline.pl" "foo\\\" bar\\\\" " +open(my $x,\">&STDOUT\");" ( "showcmdline.pl", "foo\\\" bar\\\\", "open(my \$x,\">&STDOUT\");", )