http://qs1969.pair.com?node_id=11138504

LanX has asked for the wisdom of the Perl Monks concerning the following question:

Hi

I'm doing some masstests with hundreds of thousands of random cases.

Is there an easy switch to make Test::More only report fails and stay silent with passes?

I could of course do those tests manually, but it would be nice to use the same test-suite against small number of manually generated cases.

Cheers Rolf
(addicted to the Perl Programming Language :)
Wikisyntax for the Monastery

Replies are listed 'Best First'.
Re: Test::More to only report fails?
by hippo (Bishop) on Nov 06, 2021 at 16:33 UTC

    I have a test suite which runs daily and I obviously don't want to be bothered hearing from it if everything passes :-). To that end I have used the facilities of TAP::Harness so that it only reports failures. Here's the (freshly anonymised) code:

    use strict; use warnings; use TAP::Harness; use Data::Dumper; chdir "$ENV{HOME}/src/mymods/Foo" or die "Cannot access start dir"; my $harness = TAP::Harness->new ({verbosity => -3, lib => [ qw/lib/ ]} +); my $res = $harness->runtests (glob ('t/*.t')); exit 0 if $res->all_passed; # Problems my $fail_count = $res->failed; my $total_run = $res->total; print "Failed $fail_count out of $total_run tests run.\n"; print "\tFailed: $_\n" for $res->failed;

    This has been running daily from cron for a couple of years. Not quite what you asked for but perhaps a feasible alternative. HTH.


    🦛

Re: Test::More to only report fails?
by choroba (Cardinal) on Nov 06, 2021 at 16:39 UTC
    What about using TAP::Formatter::JUnit and then use
    /testsuites/testsuite[@failures!=0]/testcase[failure]
    to select the failed tests (tested in XML::XSH2).

    map{substr$_->[0],$_->[1]||0,1}[\*||{},3],[[]],[ref qr-1,-,-1],[{}],[sub{}^*ARGV,3]
      is that similar to hippo's solution, in that it generates a long list of tests and filters the failed once a posteriori?

      In my case I'm randomly generating 1e6+ cases ...

      Cheers Rolf
      (addicted to the Perl Programming Language :)
      Wikisyntax for the Monastery

Re: Test::More to only report fails?
by jo37 (Deacon) on Nov 07, 2021 at 11:47 UTC

    Lately I came across a similar issue. In my case tests were simple boolean values and I so struck to this construct:

    #!/usr/bin/perl use Test2::V0; for my $mod (11, 4) { grep { !($_ % $mod) and !fail "$_ failed"; } 1 .. 10 or pass "all passed mod $mod"; } done_testing; __DATA__ # Seeded srand with seed '20211107' from local date. ok 1 - all passed mod 11 not ok 2 - 4 failed # Failed test '4 failed' # at /home/jo/Programs/play-scripts/test-many-grep.pl line 6. not ok 3 - 8 failed # Failed test '8 failed' # at /home/jo/Programs/play-scripts/test-many-grep.pl line 6. 1..3

    This is quite simple but has some flaws:

    • It is restricted to boolean values, i.e. compare tools cannot be used.
    • The number of tests is unpredictable, so plan N cannot be used.
    Now that almost the same question was raised, I felt the need for something more advanced than my first attempt. I never used Test::More, so my solution is based on Test2::V0.

    For this purpose a new test tool is generated. It will execute a code block in the context of a subtest, dropping all passed assertion events. Any other event will be processed unmodified.

    #!/usr/bin/perl use Test2::V0; # This would go into a separate package: use Test2::API qw(context); use Test2::Tools::Subtest qw(subtest_streamed); sub drop_passing (&$) { my ($tests, $name) = @_; subtest_streamed $name => sub { my $ctx = context(); my $hub = $ctx->hub; my $filter = $hub->filter(sub { my ($hub, $event) = @_; return if $event->facet_data->{assert}{pass}; $event; }); $tests->(); $hub->unfilter($filter); $ctx->pass($name) unless $hub->failed; $ctx->done_testing; $ctx->release; } } ### end package plan 4; pass 'pre test'; for my $mod (11, 4) { drop_passing { is $_ % $mod, T(), "check $_" for 1 .. 10; } "drop passing mod $mod"; } pass 'post test'; __DATA__ # Seeded srand with seed '20211107' from local date. 1..4 ok 1 - pre test # Subtest: drop passing mod 11 ok 1 - drop passing mod 11 1..1 ok 2 - Subtest: drop passing mod 11 # Subtest: drop passing mod 4 not ok 1 - check 4 # Failed test 'check 4' # at /usr/local/share/perl/5.20.2/Test2/API.pm line 676. # +-----+--------+-------+-----+ # | GOT | OP | CHECK | LNs | # +-----+--------+-------+-----+ # | 0 | TRUE() | TRUE | 34 | # +-----+--------+-------+-----+ not ok 2 - check 8 # Failed test 'check 8' # at /usr/local/share/perl/5.20.2/Test2/API.pm line 676. # +-----+--------+-------+-----+ # | GOT | OP | CHECK | LNs | # +-----+--------+-------+-----+ # | 0 | TRUE() | TRUE | 34 | # +-----+--------+-------+-----+ 1..2 not ok 3 - Subtest: drop passing mod 4 # Failed test 'Subtest: drop passing mod 4' # at /home/jo/Programs/play-scripts/test-many.pl line 25. ok 4 - post test

    Now any tests may be used within drop_passing and these count as a single test.

    Though this won't be of much help for users of Test::More I'd like to share it anyway.

    Greetings,
    -jo

    $gryYup$d0ylprbpriprrYpkJl2xyl~rzg??P~5lp2hyl0p$
Re: Test::More to only report fails?
by stevieb (Canon) on Nov 06, 2021 at 15:56 UTC

    This is a great question which I hope someone has a solution for.

    I think about this often, but never took the time to look into it.

Re: Test::More to only report fails?
by LanX (Saint) on Nov 06, 2021 at 19:53 UTC
    OK, after some digging :)
    use strict; use warnings; use Test::More; Test::More->builder->output(\ my $scalar); is($_,$_,"equality $_") for 1..10; is(1,0,"FAIL"); done_testing;

    C:/Strawberry/perl/bin\perl.exe -w d:/tmp/pm/redirect_Tests.pl # Failed test 'FAIL' # at d:/tmp/pm/redirect_Tests.pl line 11. # got: '1' # expected: '0' # Looks like you failed 1 test of 11.

    ... but it will still fill up $scalar with the passed tests.

    Does Perl have a generic /dev/null filehandle to ignore output like on linux?

    edit

    this works on Win, but it's not generic

    open my $null,">","nul"; Test::More->builder->output($null);

    update

    Ah yes IO::Null is on cpan. :)

    Cheers Rolf
    (addicted to the Perl Programming Language :)
    Wikisyntax for the Monastery

      hmm ... trying out IO::Null fails

      use strict; use warnings; use Test::More; use IO::Null; use Data::Dump; my $fh = IO::Null->new; Test::More->builder->output($fh); is($_,$_,"equality $_") for 1..5; is(1,0,"FAIL"); done_testing;

      C:/Strawberry/perl/bin\perl.exe -w d:/tmp/pm/redirect_Tests.pl print() on unopened filehandle GLOB at C:/Strawberry/perl/lib/Test2/Fo +rmatter/TAP.pm line 156. print() on unopened filehandle GLOB at C:/Strawberry/perl/lib/Test2/Fo +rmatter/TAP.pm line 156. print() on unopened filehandle GLOB at C:/Strawberry/perl/lib/Test2/Fo +rmatter/TAP.pm line 156. print() on unopened filehandle GLOB at C:/Strawberry/perl/lib/Test2/Fo +rmatter/TAP.pm line 156. print() on unopened filehandle GLOB at C:/Strawberry/perl/lib/Test2/Fo +rmatter/TAP.pm line 156. print() on unopened filehandle GLOB at C:/Strawberry/perl/lib/Test2/Fo +rmatter/TAP.pm line 125. # Failed test 'FAIL' # at d:/tmp/pm/redirect_Tests.pl line 13. # got: '1' # expected: '0' print() on unopened filehandle GLOB at C:/Strawberry/perl/lib/Test2/Fo +rmatter/TAP.pm line 125. # Looks like you failed 1 test of 6. Compilation exited abnormally with code 1 at Sat Nov 6 22:35:40

      Cheers Rolf
      (addicted to the Perl Programming Language :)
      Wikisyntax for the Monastery

        I get the same results on Active State:
        print() on unopened filehandle GLOB at C:/Perl64/site/lib/Test2/Format +ter/TAP.pm line 156. print() on unopened filehandle GLOB at C:/Perl64/site/lib/Test2/Format +ter/TAP.pm line 156. print() on unopened filehandle GLOB at C:/Perl64/site/lib/Test2/Format +ter/TAP.pm line 156. print() on unopened filehandle GLOB at C:/Perl64/site/lib/Test2/Format +ter/TAP.pm line 156. print() on unopened filehandle GLOB at C:/Perl64/site/lib/Test2/Format +ter/TAP.pm line 156. print() on unopened filehandle GLOB at C:/Perl64/site/lib/Test2/Format +ter/TAP.pm line 125. # Failed test 'FAIL' # at C:\Users\xxxx\Documents\PerlProjects\Monks\test.pl line 13. # got: '1' # expected: '0' print() on unopened filehandle GLOB at C:/Perl64/site/lib/Test2/Format +ter/TAP.pm line 125. # Looks like you failed 1 test of 6. Process completed with exit code 1
        With:
        use strict; use warnings; use Test::More; use IO::Null; use Data::Dump; open (my $fh, '>', 'nul') or die "Ooops$!"; Test::More->builder->output($fh); is($_,$_,"equality $_") for 1..5; is(1,0,"FAIL"); done_testing; __END__ # Failed test 'FAIL' # at C:\Users\mmtho\Documents\PerlProjects\Monks\test.pl line 13. # got: '1' # expected: '0' # Looks like you failed 1 test of 6.
        I guess just check O/S and open either /dev/null or nul.
        NUL is a reserved file name on Windows.
Re: Test::More to only report fails? (SOLVED)
by LanX (Saint) on Nov 07, 2021 at 09:22 UTC
    One solution:
    use strict; use warnings; use Test::More; tie my $s_null,"NULL"; Test::More->builder->output(\ $s_null); is($_,$_+ int rand 2,"TEST $_") for 1..5; done_testing; package NULL; sub FETCH {""} sub STORE { print $_[1] if $_[1] =~ /^not ok \d+ -/; # NB: you ma +y want to another FH } sub TIESCALAR { return bless \ my $var, shift; }

    C:/Strawberry/perl/bin\perl.exe -w d:/tmp/pm/test_only_fails.pl not ok 2 - TEST 2 # Failed test 'TEST 2' # at d:/tmp/pm/test_only_fails.pl line 11. # got: '2' # expected: '3' # Looks like you failed 1 test of 5.

    Cheers Rolf
    (addicted to the Perl Programming Language :)
    Wikisyntax for the Monastery