songmaster has asked for the wisdom of the Perl Monks concerning the following question:

I maintain a non-Perl build system that uses APP::Prove for running tests and displaying the results, with our own code for generating the TAP output from our C/C++ test programs. Those test programs are each exec'd from their own wrapper script (written in Perl) and exit with a non-zero status if any of the tests fail (this exit status is needed for other features of our build/test system). Is there an easy way to suppress the "Dubious" message that gets displayed when a non-zero status is returned, as shown below?

$ prove --failures --ignore-exit epicsTypesTest.t
epicsTypesTest.t .. 1/10 
not ok  1 - sizeof(epicsInt8) == 10
epicsTypesTest.t .. Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/10 subtests 

Test Summary Report
-------------------
epicsTypesTest.t (Wstat: 256 Tests: 10 Failed: 1)
  Failed test:  1
  Non-zero exit status: 1
Files=1, Tests=10,  0 wallclock secs (0.03 usr + 0.01 sys = 0.04 CPU)
Result: FAIL

[download]

Note that I'm already passing the --ignore-exit flag, but it doesn't seem to make any difference to what gets displayed. My App::Prove and TAP::Harness both say they are version 3.4.2. The manpage for prove says:

             --ignore-exit     Ignore exit status from test scripts.

[download]

and TAP::Parser's perldoc says:

  "ignore_exit"
      $parser->ignore_exit(1);

    Tell the parser to ignore the exit status from the test when
    determining whether the test passed. Normally tests with non-zero
    exit status are considered to have failed even if all individual
    tests passed. In cases where it is not possible to control the
    exit value of the test script use this option to ignore it.

[download]

Thanks for any insight or hints!

Replies are listed 'Best First'.
Re: Is prove's --ignore-exit flag incomplete or ignored?
by hippo (Archbishop) on Dec 18, 2020 at 22:36 UTC

    Works for me:

    $ cat exit.t
use strict;
use warnings;

use Test::More tests => 1;

is 1, 1, 'Tickety-boo';
exit 1;
$ prove --failures --ignore-exit exit.t 
exit.t .. 1/1 # Looks like your test exited with 1 just after 1.
exit.t .. ok   
All tests successful.
Files=1, Tests=1,  0 wallclock secs ( 0.02 usr  0.00 sys +  0.02 cusr 
+ 0.00 csys =  0.04 CPU)
Result: PASS
$

    [download]

    If this isn't what you expect (or what you see) please provide an SSCCE.

    🦛

[reply]
[d/l]

      Thanks for answering, how's this:

      $ cat exit.t
use strict;
use warnings;

use Test::More tests => 1;

is 1, 0, 'Tickety-boo';
exit 1;

      [download]

      which gives:

      $ prove --failures --ignore-exit exit.t
exit.t .. 1/1 
not ok 1 - Tickety-boo

#   Failed test 'Tickety-boo'
#   at exit.t line 6.
#          got: '1'
#     expected: '0'
# Looks like your test exited with 1 just after 1.
exit.t .. Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1 subtests 

Test Summary Report
-------------------
exit.t (Wstat: 256 Tests: 1 Failed: 1)
  Failed test:  1
  Non-zero exit status: 1
Files=1, Tests=1,  0 wallclock secs ( 0.03 usr  0.01 sys +  0.07 cusr 
+ 0.01 csys =  0.12 CPU)
Result: FAIL

      [download]

      The only difference between your code and my version is that I have a failing test as well a non-zero exit status, and with that the "Dubious" message appears. Omitting the --ignore-exit flag doesn't change my output, so it's not clear that the flag is actually doing anything at all.

[reply]
[d/l]
[select]

        I think you'll find that --ignore-exit is indeed ignoring the exit value; however, it is not ignoring failed tests.

        In "... our own code for generating the TAP output ...", you'll probably need to implement something along the lines of TODO or SKIP that Test::More provides which, in your code, ignores failed tests where that would be a valid thing to do.

        Consider these four runs: 

        $ for i in exit_0_test_todo.t exit_1_test_todo.t; do cat $i; echo "---
+-------------"; prove --failures $i; echo "============"; prove --fai
+lures --ignore-exit $i; echo "============"; done
use strict;
use warnings;

use Test::More tests => 1;

TODO: {
    local $TODO = 'WIP';
    is 1, 0, 'Tickety-boo';
};

exit 0;
----------------
exit_0_test_todo.t .. ok
All tests successful.
Files=1, Tests=1,  0 wallclock secs ( 0.01 usr  0.01 sys +  0.03 cusr 
+ 0.06 csys =  0.12 CPU)
Result: PASS
============
exit_0_test_todo.t .. ok
All tests successful.
Files=1, Tests=1,  0 wallclock secs ( 0.02 usr  0.02 sys +  0.03 cusr 
+ 0.06 csys =  0.12 CPU)
Result: PASS
============
use strict;
use warnings;

use Test::More tests => 1;

TODO: {
    local $TODO = 'WIP';
    is 1, 0, 'Tickety-boo';
};

exit 1;
----------------
exit_1_test_todo.t .. 1/1 # Looks like your test exited with 1 just af
+ter 1.
exit_1_test_todo.t .. Dubious, test returned 1 (wstat 256, 0x100)
All 1 subtests passed

Test Summary Report
-------------------
exit_1_test_todo.t (Wstat: 256 Tests: 1 Failed: 0)
  Non-zero exit status: 1
Files=1, Tests=1,  1 wallclock secs ( 0.03 usr  0.02 sys +  0.03 cusr 
+ 0.06 csys =  0.14 CPU)
Result: FAIL
============
exit_1_test_todo.t .. 1/1 # Looks like your test exited with 1 just af
+ter 1.
exit_1_test_todo.t .. ok
All tests successful.
Files=1, Tests=1,  0 wallclock secs ( 0.03 usr  0.00 sys +  0.06 cusr 
+ 0.03 csys =  0.12 CPU)
Result: PASS
============

        [download]

        — Ken

[reply]
[d/l]
[select]

Back to Seekers of Perl Wisdom