(Update: Solved to my satisfaction for this particular problem. New runner in comment below.)

I have a big test that has quite a few configuration options and reasons to run subtests or not depending on environment. Works great but it cannot play well with the master runner because there are a couple hundred other tests that are mostly without the general environment concerns and none with the same granularity of configuration.

I want to run the test with three (or more) configurations/environments. I can certainly just do that inside the test. It’s trivial. However! It’s already a huge test, I don’t know that my current permutations are final, it’s confusing enough already and another layer or two of indentation and wrapping all of the subtests… it might be trivial to write, it will not be fun to read or edit, and it won’t be as clean to exit, skip, rerun, run alone/once with a single set of the permutations as it is now…

So… This is a summary of my DWIM first idea, but not something that actually does it –

run.t.raw, the original test

#!perl
use strictures;
use Test::More;

subtest "Prod" => sub {
    plan skip_all => "Set PROD_TEST to run" unless $ENV{PROD_TEST};
    ok 1, "OHAI, PROD!";
    done_testing(1);
};

subtest "Dev" => sub {
    plan skip_all => "Set DEV_TEST to run" unless $ENV{DEV_TEST};
    ok 1, "OHAI, DEV!";
    done_testing(1);
};

done_testing(2);

[download]

runner.t

#!perl
# File: runner.t
use strictures;
use App::Prove;

for my $env (qw/ PROD DEV /)
{
    my $test = join "_", $env, "TEST";
    local $ENV{$test} = 1;
    my $app = App::Prove->new;

    $app->process_args( -I => "./",
                        -v => "run.t.raw" );

    eval { $app->run } or die "Failed... not proceeding: $@";
}

[download]

Without runner / normal prove

prove -v run.t.raw
run.t.raw .. 
# Subtest: Prod
    1..0 # SKIP Set PROD_TEST to run
ok 1 # skip Set PROD_TEST to run
# Subtest: Dev
    1..0 # SKIP Set DEV_TEST to run
ok 2 # skip Set DEV_TEST to run
1..2
ok
All tests successful.
Files=1, Tests=2,  0 wallclock secs ( 0.02 usr  0.01 sys +  0.06 cusr 
+ 0.01 csys =  0.10 CPU)
Result: PASS

[download]

How it falls down / fails to DWIM

prove runner.t 
runner.t .. All 2 subtests passed 
        (less 2 skipped subtests: 0 okay)

Test Summary Report
-------------------
runner.t (Wstat: 0 Tests: 6 Failed: 4)
  Failed tests:  1-3, 6
  Parse errors: Plan (1..2) must be at the beginning or end of the TAP
+ output
                Tests out of sequence.  Found (1) but expected (4)
                Tests out of sequence.  Found (2) but expected (5)
                More than one plan found in TAP output
                Bad plan.  You planned 2 tests but ran 6.
Files=1, Tests=6,  0 wallclock secs ( 0.01 usr  0.01 sys +  0.18 cusr 
+ 0.05 csys =  0.25 CPU)
Result: FAIL

[download]

Sidenote, works perfectly but cannot be run with prove or in normal harness

perl runner.t
# …
All tests successful.

[download]

My question: How can I make the “runner” act like a single “master” test in a clean way? I may end up doing it in a messy way but I feel certain that I’m just missing some simple idea. I tried some do and straight eval stuff but without getting kind of complicated, it won’t do the right thing because it issues too many start/stop/count statements for the runner to accept it as one “master” test.

In reply to Test runner that acts like a test and can be run as one by Your Mother

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.