I have two questions here, one about what I want to do, the other about what I'm currently doing.

I'm writing tests for an early-stage mathematical proof checker (axiom). In some cases, there are tests for which there are multiple results that would class as "correct", but some results are better than others.

For example, when stringifying a mathematical expression such as a-b, the result 'a(-b)' would be "not ok", but any of 'a+(-b)', 'a+-b', 'a-b' would be "ok" - and with some more ok than others.

Has anybody implemented a test regime that accounts for this sort of thing? The only approach I've thought of so far is some sort of "quality" metric, independent of pass/fail statistics, wherein "more ok" results push the quality metric up. It is not obvious to me how to quantify that though, or how results from individual tests should combine.

Secondly, I'm currently trying to implement these as TODO tests, using Test::More and prove. However when I run a test file individually, a failing TODO test stands out like a true fail; and when I run under prove it just gives output like:

t/expr-stringify.t .. ok    
All tests successful.
Files=1, Tests=29,  0 wallclock secs ( 0.03 usr  0.00 sys +  0.31 cusr
+  0.04 csys =  0.38 CPU)
Result: PASS

[download]

.. which gives no indication that there were failing TODO tests.

What are people's preferred way to get a clean indication of "all passing, but with $n failing TODO tests"?

In reply to Testing: shades of grey by hv

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.