Testing and system return status

neilwatson has asked for the wisdom of the Perl Monks concerning the following question:

Greetings,

I'm sure I've made a logic mistake somewhere, but I can't see where. I'm testing a command, sending it invalid inputs that will cause it to exit none zero. My test does not seem to work. Here's a snippet to reproduce the effect.

subtest 'Invalid CLI input' => sub
{
   my @args;
   for my $key ( keys %{ $query_params } )
   {
      next if ( $key eq 'report_title' );
      push @args, '--'.$key, "\'$query_params->{$key}\'";
   }

   my $stdout  = '/tmp/'.$$.'_stdout';
   my $command = './script/query '. join( ' ', @args ) ." &>$stdout";

   ok ( system( $command ), 'CLI should return none zero status' ) or
   warn "not ok CLI query command [$command] did NOT return false [$?]
+";

}

The output:
    not ok 1 - CLI should return none zero status
    #   Failed test 'CLI should return none zero status'
    #   at ./test.pl line 62.

not ok CLI query command [./script/query --delta_minutes '; DROP
TABLES' --latest_record '0' --hostname 'ettin;' --policy_server ';
DELETE FROM' --class 'any;' --gmt_offset '\;400; EXIT' --ip_address
'10.com' --timestamp '$400;' &>/tmp/32531_stdout] did NOT return false
[0] at ./test.pl line 62.

    1..1
    # Looks like you failed 1 test of 1.
not ok 1 - Invalid CLI input
#   Failed test 'Invalid CLI input'
#   at ./test.pl line 84.
# Tests were run but no plan was declared and done_testing() was not
# seen.
[download]

$command should return 1, an error, and thus the ok test should be true. Yet it seems to fail and the return value, $?, is zero. If I run the command by hand, in the shell. What have I missed?

 $ ./script/query --delta_minutes '; DROP TABLES' --latest_record '0'
 --hostname 'ettin;' --policy_server '; DELETE FROM' --class 'any;'
 --gmt_offset '\;400; EXIT' --ip_address '10.com' --timestamp
 '$400;' &>/tmp/32531_stdout; echo $?
1
[download]

Update

Two fixes. First &> did not redirect. I had to use the longer 2>&1 1>. Second, the return status was due to a wrapper around the query command. Once is fixed the wrapper to capture and return the exit status, the test passed.

Neil Watson
watson-wilson.ca

Comment on Testing and system return status Select or Download Code

Replies are listed 'Best First'.
Re: Testing and system return status by Ea (Chaplain) on Sep 10, 2014 at 14:11 UTC
Just a thought. How about changing the `ok(system())` to `is(system(), 1)` to see what the system call is actually returning? It could conceivably be as maddening as to return the success of your redirecting the output. Sometimes I can think of 6 impossible LDAP attributes before breakfast.	[reply] [d/l] [select]
Re^2: Testing and system return status (was 0, shell) by tye (Sage) on Sep 10, 2014 at 14:45 UTC
I think that is a fine suggestion. But note that the information in question is already available: `'10.com' --timestamp '$400;' &>/tmp/32531_stdout] did NOT return false [0] at ./test.pl line 62.` [download] The "`[0]`" part shows that the prior system() returned 0 (unless some weird thing is invoking some other child process in between). For the original question, I would examine the contents of the /tmp/...stdout files in the two cases. Also note that this invocation of system will actually invoke a shell so that it can parse the command line and it will be the shell's exit value that system() will report. So my guess is that the shell is not properly propagating the child's success/failure. - tye	[reply] [d/l] [select]
Re^3: Testing and system return status (was 0, shell) by Ea (Chaplain) on Sep 11, 2014 at 13:58 UTC
You are correct. I didn't understand the significance of `$?` until you pointed it out. Sometimes I can think of 6 impossible LDAP attributes before breakfast.	[reply] [d/l]
Re: Testing and system return status by GrandFather (Saint) on Sep 10, 2014 at 02:21 UTC
You may be able to reproduce something with that "code", but as given it's not run-able. That aside, aren't you testing your process return result backwards? By convention system returns 0 for success so your ok test should be something like `ok(0 == system($command), 'CLI should return none zero status');` [download] Update sigh, coffee is not an antidote to cold induced muzzy headedness. Muzzy headed comment struck. Perl is the programming world's equivalent of English	[reply] [d/l]
Re^2: Testing and system return status (!backwards) by tye (Sage) on Sep 10, 2014 at 03:00 UTC
It would be backward if a passing test was one where the command exited with a "success" status. In this case, passing the test is the command exiting with a "failure" status [which would then pass a "true" value to ok()]. So the test is not backward. - tye	[reply]