Relating Pert test report number to test items in the code

ron7 has asked for the wisdom of the Perl Monks concerning the following question:

I've searched, but could not find an automated way of relating the test numbers output to the console by the perl test_harness (ExtUtils::Command::MM normally invoked through 'make test') to the actual tests in the *t programs; eg:

../t/10.system.t (Wstat: 1024 Tests: 82 Failed: 2)
  Failed tests:  77-78

Great! But which of 82 is 77 (without error-prone and painful manual counting)!!!

Manually inserting the number as a searchable string in a comment is going to get out of date real quick; I'm just not that disciplined.

Tracking by the string message of the failing test is practical (mostly) but painful.

I'd have though there's be a module which automatically adds a comment line with the sequential test number before each statement that Test is going to evaluate. If there is, I can't find it. How do others resolve this problem on non-trivial test suites?

Comment on Relating Pert test report number to test items in the code

Replies are listed 'Best First'.
Re: Relating Pert test report number to test items in the code by ikegami (Patriarch) on Feb 11, 2011 at 05:56 UTC
Why are you looking at the summary to find info about a specific test? The test output should have been more like `a.t .. 1/4 # Failed test at a.t line 5. # got: '0' # expected: '2' # Looks like you failed 1 test of 4. a.t .. Dubious, test returned 1 (wstat 256, 0x100) Failed 1/4 subtests Test Summary Report ------------------- a.t (Wstat: 256 Tests: 4 Failed: 1) Failed test: 3 Non-zero exit status: 1 Files=1, Tests=4, 0 wallclock secs ( 0.02 usr + 0.03 sys = 0.05 CPU +) Result: FAIL Failed 1/1 test programs. 1/4 subtests failed.` [download] Or if you gave your test a name: `a.t .. 1/4 # Failed test 'foo' # at a.t line 5. # got: '0' # expected: '2' # Looks like you failed 1 test of 4. a.t .. Dubious, test returned 1 (wstat 256, 0x100) Failed 1/4 subtests Test Summary Report ------------------- a.t (Wstat: 256 Tests: 4 Failed: 1) Failed test: 3 Non-zero exit status: 1 Files=1, Tests=4, 0 wallclock secs ( 0.01 usr + 0.00 sys = 0.01 CPU +) Result: FAIL Failed 1/1 test programs. 1/4 subtests failed.` [download] In both case, it specifies the line number ("at a.t line 5"). If you provide a test name (2nd param for `ok()`, 3rd for `is()`), it provides that too ("Failed test 'foo'"). If you can't find the test even with the exact line number and the test name, your name is probably too generic.	[reply] [d/l] [select]
Re^2: Relating Pert test report number to test items in the code by ron7 (Beadle) on Feb 14, 2011 at 08:29 UTC
Well don't I feel like a fool. All the failed tests I've looked at and never noticed/paid attention to the "# at <file> <line>" output! Thanks!	[reply]
Re: Relating Pert test report number to test items in the code by moritz (Cardinal) on Feb 11, 2011 at 08:13 UTC
The second argument to `ok()` and the third argument to `is()` etc. is a description string. So you could write: `use Test::More tests => 2; ok 1, '1 is True'; ok 0, '0 is also True'; # will fail` [download] If you run `perl t/your-test.t`, the output looks like this: `1..2 ok 1 - 1 is True not ok 2 - 0 is also True # Failed test '0 is also True' # at foo.t line 3. # Looks like you failed 1 test of 2.` [download] It includes the description of the failed test, so you can just search in the test file for that description. Perl 6 - second systems done right	[reply] [d/l] [select]
Re: Relating Pert test report number to test items in the code by Khen1950fx (Canon) on Feb 11, 2011 at 09:51 UTC
There is a module that does it: TAPx::Parser. There's a simple cmdline script that comes with it called runtests. You'll need to manually download and manually install it because some of the tests will fail; however, don't worry about that because it does work. For runtests: `runtests -v /path/to/test` [download] Or you can give a directory and run as many tests as you want: `runtests -v /path/to/tests/directory` [download]	[reply] [d/l] [select]