Re^4: Need advice on test output

Thanks for the feedback. I hope the following helps.

What is 'stat'? How does it help identify the failures?: 'stat' is the exit code and indicates how many tests failed. However, since it doesn't report numbers in excess of 255, it's not terribly useful and I don't know that it's used.
Ditto 'Wstat'?: 'wstat' is the wait status of the test. I also don't know how it's used and initially I didn't provide it, but I was told emphatically on the QA list that I should, so I did.
What does 'UNEXPECTEDLY SUCCEEDED' mean?: This is the number of tests which were marked TODO but passed anyway.
If a test is designed to fail, then does it get reported as a failure when it does fail? Or is that an 'EXPECTED FAILURE'?: A test designed to fail is generally a TODO test and if it fails, it it not reported as a failure or an 'EXPECTED FAILURE'.
Which test 'UNEXPECTEDLY SUCCEEDED'?: Currently Test::Harness is not able to track or report which tests unexpectedly succeeded but TAPx::Harness can and does.
If it's not important enough to tell me which one, why is it important enough to bother mentioning it at all?: See the note to the previous question. It is important, but Test::Harness doesn't have this ability.
What is the difference between "test scripts" and "test programs"?: Nothing.
And if they are the same thing, why is it neccesary to give me the same information twice?: I don't understand this question.
Actually, 3 times. "Files=3, Tests=33, " is just a subset of the same information above and below it.: It's a summary report. You may find it useful or you may not. Alternate suggestions welcome :)
When was the last time anyone optimised there test scripts? Is there any other use for that timing information?: I sometimes use it when I'm profiling my code and make try to optimize it. The timing information often tells me if I've made a significant difference.
Verbose output that tells me nothing useful, whilst pushing useful information off the top of my buffer is really annoying. Yes, I could redirect it to null, but then I won't see the useful stuff when something fails.: That's a good point. I could easily make a 'quiet' mode which only reports overall success or failure. That would let you rerun the test suite to see what actually failed, if anything.
Converting 5/10 into a running percentage serves no purpose.: Agreed. I was just trying to mimic the behavior of Test::Harness. Others have pointed out that it doesn't help and I'll probably just drop it.
When something fails, tell me what failed and where. Eg. File and line number. (Not test number).: Unfortunately, TAP format does not support this. Those data are embedded in the diagnostics and there is no way to disambiguate this information from the other diagnostic information. This is a feature that is planned for TAP 2.0.

I might add that the runtests utility I've written (not yet on the CPAN but analogous to prove), allows you to specify which test harness you want to run your tests through. Thus, you can easily create a new harness to satisfy your particular needs.

Cheers,
Ovid

New address of my CGI Course.

Comment on Re^4: Need advice on test output

Replies are listed 'Best First'.
Re^5: Need advice on test output by BrowserUk (Patriarch) on Jan 05, 2007 at 14:37 UTC
I don't understand this question. In the output posted, the following three lines appear consecutively: `Failed 2/3 test scripts. 5/33 subtests failed. Files=3, Tests=33, 0 wallclock secs ( 0.10 cusr + 0.01 csys = 0.11 +CPU) Failed 2/3 test programs. 5/33 subtests failed.` [download] The first and last are identical except for the words "scripts" and "programs". Why repeat the same information? And the first two fields of the middle line, "Files=3" replicates the last digit of "Failed 2/3 test scripts". "Tests=33" replicates the last digits of "5/33 subtests failed" It's a summary report. You may find it useful or you may not. 2 1/2 of those 3 lines are redundant. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. "Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."	[reply] [d/l]

Replies are listed 'Best First'.

Re^5: Need advice on test output
by BrowserUk (Patriarch) on Jan 05, 2007 at 14:37 UTC

I don't understand this question.

In the output posted, the following three lines appear consecutively:

Failed 2/3 test scripts. 5/33 subtests failed.
Files=3, Tests=33,  0 wallclock secs ( 0.10 cusr +  0.01 csys =  0.11 
+CPU)
Failed 2/3 test programs. 5/33 subtests failed.
[download]

The first and last are identical except for the words "scripts" and "programs". Why repeat the same information?

And the first two fields of the middle line,

"Files=3" replicates the last digit of "Failed 2/3 test scripts".
"Tests=33" replicates the last digits of "5/33 subtests failed"

It's a summary report. You may find it useful or you may not.

2 1/2 of those 3 lines are redundant.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

"Too many [] have been sedated by an oppressive environment of political correctness and risk aversion."

[reply]
[d/l]