Is it worth the work to provide default output as shown and allow configuration options to change what is shown and how it is shown?

Stimulated by a recent Q&A post I wonder if format could be pressed into service to provide user specified formatting? Combined with control over the data provided, that would allow pretty full on control over formatting, albeit in a slightly archaic fashion.

Alternatively a format string for printf could be provided by the user, again with user control over the data provided.

Both these techniques imply that multiple format strings or pictures are provided to allow control over different report line types.

A format string for 'printf' would be difficult as there are a lot of conditional bits which won't get displayed if they're not present (such as 'unexpectedly succeeded tests'). Perhaps allowing an optional Template Toolkit template might work. However, TAPx::Parser very deliberately does not use any non-core modules and strives to be compatible with versions of Perl as old as 5.005, so this also limits my options.

Cheers,
Ovid

New address of my CGI Course.

Sorry, I didn't make that very clear did I? I meant to imply a format string per default summary line type. So the table format could easily be altered with column orders specified in a list of column names for example. This works pretty well if the column information happens to be stored in a hash, but it it pretty trivial to use a hash to map from a list of names to an index in any case.

Report information that by default was suppressed would still be suppressed, but a specific format string could be supplied if required.

---

DWIM is Perl's answer to Gödel

That said, I deeply encourage you to make the intermediate abstract test result available for any further processing. How you serialize it is not so important. We've used YAML in Pugs, but Test::TAP::Model doesn't really care. Speaking of which, I've not been following your design, have you considered using that model in the first place?

The biggest problem affecting the venerable Test::Harness is that the parsing, analyzing, and presentation of TAP output has been so tightly coupled that people have found it very hard to do anything "unusual" with it. My primary goal has been to break that coupling so that you can do anything you want. So the TAPx::Harness uses the TAPx::Parser to parse and analyze results and it presents the summary. It's simple enough that you can now use anything else you want to present this information. So yes, the intermediate abstract test results are all available in a very easy to use format. So here's one simplistic way, for example, of just printing failed tests:

my $parser = TAPx::Parser->new( { source => 't/customers.t' } );
while ( defined ( my $result = $parser->next ) ) {
    print $result->as_string . "\n" if ! $result->is_ok;
}
[download]

Cheers,
Ovid

New address of my CGI Course.

I know I'm gonna regret asking, but what do people do with this output?

Does it get printed out and stuck on the wall? Or end up in the corporate report? Or archived to cd and kept for prosterity?

Or does it simply scroll up the screen and disappear into the bitbucket!

I'm not sure what you're asking. I read the output to find out which tests, if any, are failing.

If one needs this output in radically different format (one person I know needs it in XML), then they simply write a different harness which creates the output format they need. For the moment, I'm focusing on the typical "I just ran my test suite and have a bunch of stuff on the terminal." I want to know how that 'stuff' should be formatted to be most useful to you.

Cheers,
Ovid

New address of my CGI Course.

Failed Test Stat Wstat Total Fail  List of Failed
----------------------------------------------------------------------
+---------
t/bar.t        4  1024    13    4  2 6-8
t/foo.t        1   256    10    1  5
 (1 subtest UNEXPECTEDLY SUCCEEDED).
Failed 2/3 test scripts. 5/33 subtests failed.
Files=3, Tests=33,  0 wallclock secs ( 0.10 cusr +  0.01 csys =  0.11 
+CPU)
Failed 2/3 test programs. 5/33 subtests failed.
[download]

1. What is 'stat'? How does it help identify the failures?
2. Ditto 'Wstat'?
3. What does 'UNEXPECTEDLY SUCCEEDED' mean?

   If a test is designed to fail, then does it get reported as a failure when it does fail? Or is that an 'EXPECTED FAILURE'?

4. Which test 'UNEXPECTEDLY SUCCEEDED'?

   If it's not important enough to tell me which one, why is it important enough to bother mentioning it at all?

5. What is the difference between "test scripts" and "test programs"?

   And if they are the same thing, why is it neccesary to give me the same information twice?

   Actually, 3 times. "Files=3, Tests=33, " is just a subset of the same information above and below it.

6. When was the last time anyone optimised their test scripts?

   Is there any other use for that timing information?

Of course, you'll be taking my thoughts on this with a very large pinch of salt as I do not use these tools. The above are some of the minor reasons why not.

Much more important is that there are exactly two behaviours I need from a test harness.

• The default, no programming, no configuration, pull it and run it, out-of-the-box behaviour is that I point at a directory of tests and it runs them. If nothing fails, it should simply say that.

  "Nothing failed" or "All tests passed".

  I have no problem with a one line, in place progress indicator ("\r..."), but it should not fill my screen buffer with redundant "thats ok and that ok and thats ok" messages. I use my screen buffer to remember things I've just done: the results of compile attempts, greps etc.

  Verbose output that tells me nothing useful, whilst pushing useful information off the top of my buffer is really annoying. Yes, I could redirect it to null, but then I won't see the useful stuff when something fails.

  Converting 5/10 into a running percentage serves no purpose. A running percentage is only useful if it will allow me to predict how much longer the process will take. As the test harness doesn't know how many tests it will encounter up front, much less how long they will take, a percentage is just a meaningless number.

  If I really want this summary information, or other verbose information, (say because the tests are being run overnight by a scheduler and I'd like to see the summary information in the morning), I have no problem adding a command line switch (say -V or -V n) to obtain that information when I need it.

• When something fails, tell me what failed and where. Eg. File and line number. (Not test number).

  Preferably, it should tell me which source file/linenumber (not test file) I need to look at, but the entire architecture of the test tools just does not allow this, which is why I will continue to embed my tests in the file under test.

---

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."

$results = {
    'scripts' => {
        't/qrt.t' => {
            'Status' => 'OK', 'Stat' => '9', 'Wstat' => '2048',
            'Total' => '12', 'Fail => '0', 'FailList' => { }
        },
        't/bar.t' => {
            'Status' => 'Failed', 'Stat' => '4', 'Wstat' => '1024', 
            'Total' => '13', 'Fail => '4', 'FailList' => {
                '2' => 'Failed msg 2', '6' => 'Failed msg 6',
                '7' => 'Failed msg 7', '8' => 'Failed msg 8'
            }
        },
        't/foo.t' => {
            'Status' => 'Failed', 'Stat' => '1', 'Wstat' => '256', 
            'Total' => '10', 'Fail => '1', 'FailList' => {
                '5' => 'Unexpectedly Succeeded'
            }
        }
    },
    'time' => '0 wallclock secs ( 0.10 cusr + 0.01 csys = 0.11 CPU)'
};
[download]

I removed percentages because they're a false indication. The percentage of tests that pass are irrelevant. They either pass or they fail. There's not a sort of "degree of passing" in a test suite, or shouldn't be.

xoxo,
Andy