But why is there a need to know the exact amount of tests beforehand? Someone mentioned catching dieing or exits, but that should be easy enough to detect.

Not so easy :-)

Normal exceptions, exiting with a non-zero value, etc. are detected. Unexpected crashes from XS code, POSIX::_exit, or exit(0) are not. See below for why.

. I suppose it's all down to what philosophy lies behind the test systems from the start: most other systems I've seen have setup and teardown, so it is easy to tell if eveything ran (died after test # 12).

It's not really that aspect that causes the problem. Indeed Perl has a couple of testing frameworks that use the xUnit model (Test::Unit and my Test::Class).

The issue is that in frameworks like JUnit, TestNG, etc. the test runner and the tests are running in the same process. If the tests exit for an unexpected reason then nothing runs - so hard crashes of the system are trivial to detect (hey - my test suite has crashed!)

In Perl the test runner (Test::Harness) and the tests (the *.t files) run in separate processes.

This is good in many ways, because it makes our tests more flexible (e.g. we can run tests in different programming languages) and has the advantage that a hard crash will not stop all the other tests running.

The disadvantage is that the test runner has to figure out for itself whether that test script exited because all the tests had run, or whether it exited because something unexpected happened.

One way it does this is via the fixed plan. Double checking that the number of tests that were expected to run actually ran.

It seems very un-lazy to have to run around updating that number all the time, counting properties manually and so on. I do not see the win.

The win is that we have a more flexible testing framework by splitting the test runner and the test producer up.

The cost is that detecting unexpected failures is harder. There's been some conversation on the perl-qa list recently on just this topic, with some ideas that might make the 'no_plan' option even safer.

In reply to Re^4: Toggling test plans with vim by adrianh
in thread Toggling test plans with vim by Ovid

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.