jtroy has asked for the wisdom of the Perl Monks concerning the following question:

I can't seem to figure out how to properly include double-quote characters inside arguments to an external program while capturing both the normal and error output from that program.

I have a text file called arguments.txt that contains the following lines (with an empty line at the end).

Test
Hi (12" mix)
test.txt

I have a program called test_args.pl that reads in the list of words from the arguments.txt file. It then calls an external program using each word as an argument and captures both the normal and error output. For this test, it simply prints the output.

### test_args.pl ###
use strict;

my @args;
open (IN, "arguments.txt") or die;
while (<IN>)
{
    chomp;
    s|"|\\"|g;
    push @args, qq("$_");
}
close (IN);

my $arg_str = join(" ", @args);
my $output = `print_args.pl $arg_str 2>&1`;
print "Output:\n$output";

[download]

The external program, print_args.pl, prints to STDERR each argument it is passed on a separate line:

### print_args.pl ###
use strict;

for (my $i=0; $i<=$#ARGV; $i++)
{
    print STDERR "Arg $i - $ARGV[$i]\n";
}

[download]

When I run test_args.pl, I get the following output:

Arg 0 - Test
Arg 1 - Hi (12" mix)
Arg 2 - test.txt
Arg 3 - 2>&1
Output:

So Perl is interpreting the 2>&1 as an argument instead of redirecting stderr to stdout for the external program call, which is what I want. It fails to capture the output of print_args.pl since it's being written to sderr.

However, if I remove the " from the "Hi (12" mix)" argument, it interprets it correctly:

Output:
Arg 0 - Test
Arg 1 - Hi (12 mix)
Arg 2 - test.txt

Even stranger to me is that, if there are an even number of double-quotes in all of the parameters, it is interpreted correctly:

Output:
Arg 0 - Tes"t
Arg 1 - Hi (12" mix)
Arg 2 - test.txt

It's only when there are an odd number of double-quotes that Perl interprets 2&>1 as an argument. It's confusing because each argument seems to be interpreted correctly - the double-quotes just mess up the redirection.

I have tried this on both of my machines:

Win2k running Perl 5.8.4, ActiveState binary build 810
WinXP running Perl 5.6.1, ActiveState binary build 635

Can anyone shed some light on this for me? Am I doing something wrong? Could there be a bug in my versions of Perl? Or could it be an operating system issue?

Replies are listed 'Best First'.
Re: External Program Arguments Problem
by Stevie-O (Friar) on Dec 22, 2004 at 03:09 UTC
    I believe that Perl uses cmd.exe under NT kernels for its shell.

    Unfortunately, '\"' is not the correct way to escape a quote symbol in the command line. '^"' is.

    The world of Win32 gets a little weird when you start to deal with running other processes.

    --Stevie-O 
    $"=$,,$_=q>|\p4<6 8p<M/_|<('=>
.q>.<4-KI<l|2$<6%s!<qn#F<>;$,
.=pack'N*',"@{[unpack'C*',$_]
}"for split/</;$_=$,,y[A-Z a-z]
         {}cd;print lc

    [download]
[reply]
[d/l]
      One can also use Win32::Process::Create .
      Perl 5.8 has Win32::Job
      It rocks since you can specify filehandles for redirecting the output of the spawned process.

[reply]

        Ahh, thanks Stevie-O. This does seem to be a problem with cmd.exe - I get the same results when I use it directly. Unfortunately, ^" doesn't seem to work correctly, nor does "". Very frustrating. Is there any way to compile Perl on my Windows environment to use the Cygwin bash shell instead?

        That looks interesting, ZlR. I'll check out those modules and see if I can easily change my program to use them. I wasn't really looking to redirect the output to a filehandle, though -- just capture it in a variable. I'm calling an external program to set the music tags on MP4 files that I'm processing, so I wanted to capture its error messages in case the program failed. This problem arose for some song names and artists that contained a " (usually 12" mix or something similar).

[reply]

      Okay, Google'd and tested out a few things. It turns out I can get it to work in cmd.exe by replacing " with "^"" (all four characters). This works both from the cmd.exe shell and calling it from Perl:

      #### arg_test.pl ####
my $output = `print_args.pl "Test" "Hi (12"^"" mix)" "test.txt" 2>&1`;
print "$output";

      [download]
      C:\scripts>print_args.pl "Test" "Hi (12"^"" mix)" "test.txt" 2>&1
Arg 0 - Test
Arg 1 - Hi (12" mix)
Arg 2 - test.txt

c:\scripts>arg_test.pl
Arg 0 - Test
Arg 1 - Hi (12" mix)
Arg 2 - test.txt

      However, this isn't consistent for other external programs. I wrote a simple C program to do the same thing as print_args.pl. Replacing " with "^"" in parameters still works from the command line, but now it fails when running it from Perl.

      // PrintArgs.exe
#include <stdio.h>
int main(int argc, char **argv)
{
    for (int i=0; i<argc; i++)
    {
        fprintf(stderr, "Arg %d: %s\n", i, argv[i]);
    }
    return 0;
}

      [download]
      #### arg_test.pl ####
my $output = `PrintArgs.exe "Test" "Hi (12"^"" mix)" "test.txt" 2>&1`;
print "$output";

      [download]

      Here's the output I get running it directly in the shell and then from Perl:

      C:\scripts\arg_problem>PrintArgs.exe "Test" "Hi (12"^"" mix)" "test.txt" 2>&1
Arg 0: PrintArgs.exe
Arg 1: Test
Arg 2: Hi (12" mix)
Arg 3: test.txt

C:\scripts\arg_problem>perl_test.pl
Arg 0: PrintArgs.exe
Arg 1: Test
Arg 2: Hi (12^
Arg 3: mix) test.txt 2>&1

      This is killing me. How is the Perl-shell interaction fine in the first case but not the second? Is Perl using the exact shell on my system (c:/windows/system32/cmd.exe) or its own version? Could it be processing the parameters in some way before passing them off to the shell?

      I'd really appreciate any ideas on what's going on.

[reply]
[d/l]
[select]

Back to Seekers of Perl Wisdom