File and line number makes no sense in the context of a failed test.

Oh contrare. I'd at least have a starting point, even in this somewhat contrived example.

In more normal cases, about 95% of those test scripts I've looked at, that consist of long linear lists of unnumbered ok()s and nok()s, having the line number of the failing test would save me from have to to play that most rediculous of games--count the tests. Are they numbered from zero or one? Does a TODO count or not? Do tests that exists inside runtime conditional if blocks count if the runtime condition fails? If no, how can I know whether that runtime condition was true of false? Etc.

Of course, in this case I'd need other information too. But then in this case, the test number would be of no direct benefit either. In this case I'd have to modify the .t file to print out a sorted list of the keys to %tests at runtime, as there would be no other way to work out which test related to test N.

Oh damn! But then tracing stuff out from with in a test script is a no-no, because the test tools usurp STDOUT and STDERR for their own purposes, taking away the single most useful, and most used, debugging facility known to programmer kind: print.

And there you have it, todays number one reason I do not use these artificial, overengineered, maniacally OO test tools. They make debugging and tracing the test script 10 times harder than doing so for the scripts they are meant to test.

They are an unbelievably blunt instrument, who's basic purpose is to display and count the numbers of boolean yeses and nos. To do this simple function

• they usurp the simplest and best debugging tool available.
• force me to divorce my tests from the code under test.
• curtail my ability to use debuggers.
• wrap several layers of complication around the debugging process.
• and throw away reams of useful--I would say vital--information in the process.

And all of this so as to produce a bunch of 'pretty pictures and statistics' that I have no use for and have to use yet another layer (the test harness) to sift and filter to produce the only statistic I am interested in.

What failed and where?

For all the world this reminds me of those food ads and packaging that proclaim to world; "Product X is 95% fat free!". Ug. You mean that 5% of that crap is fat?

To date, the best testing tool I've seen available is Smart::Comments. It's require, assert, ensure, insist, check, confirm, & verify methods are amazingly simple, amazingly powerful.

• These allow me to place the tests right there in the code being tested.
• One file including code and tests.
• When failures occur, I get useful information, including but not limited to the file and line number where the failing test occurred.
• They are easily and quickly enabled & disabled by the addition of a single comment card at the top of the code.
• I can enable them on a per file basis and so only test that code I am interested in and not wait for all the tests I'm not interested in to execute first.
• I can have multiple levels of test that allow me to use a course granularity of tests to zone in on the failure and then fine granularity to isolate the exact point of failure--in the code that is being tested, not some third party test script.
• Most importantly of all. They allow me to take some users testcase that is causing failures in my code, turn the tracing and debugging on in my module(s), run that user script, and see the results.

  This is immediate and accurate.

  I do not have to modify the user supplied testcase in any way. And that is the holy grail of testing. Run the user script, unmodified on my system with debugging enabled within my modules only.

  And if the users testcase has a bunch of complex dependancies that I do not or cannot have, I can instruct the user to go into his copy of my modules and delete 1 character and all of my tests are enabled. He can then run his testcase in his environment and supply the output to me, and I can see exactly what went on.

  This is priceless!

• Finally, when testing is complete, being a source filter, commenting out the use line, means that all--every single bit of the test code; the overhead; the setup; everything--gets the hell outta dodge. It is simply gone.

Smart::Comments is the single, most useful, and most underrated module that theDamian (and possibly anyone) has yet posted to CPAN. I recognised the usefulness of the concept long ago when I came across Devel::StealthDebug, which may or may not have been the inspiration for Smart::Comments. In use, the former proved to be somewhat flaky, but theDamian has worked his usual magic with the concept (whether it was the inspiration for it or not), and come up with a real, and as yet unrecognised, winner.

To achieve the perfect test harness,

1. supply a patch to Smart::Comments that allows it to be enabled/disabled via an environment variable, (with the default being OFF).
2. Also patch it so that with an appropriate setting in that environment variable, failing asserts et al, becomes non-fatal. So that they log the assertion failure (warn style, but with Carp::cluck-style traceback), and allow the code to continue (as it would in production environments).

   The information, as logged for failure, would also be logged for success in this mode of operation.

3. Write a test harness application to parse that output and produce whatever statistics and summary information is useful.

Why haven't I written it yet? Because I keep hoping that Perl6, oops, Perl 6 is 'just around the corner', and I'm hoping that Smart::Comments will be built-in.

Of course, a few additional modules wouldn't go amiss. Smart::Comments::DeepCompare, Smart::Comments::LieToTheCaller and few others, but mostly it's all right there.