emalossi has asked for the wisdom of the Perl Monks concerning the following question:

Greetings Monks,
First post here, I hope you can nudge me in the direction of an answer to this question. I'm building a test framework that tests a number of different components, each with it's own suite of test cases. So far I've been able to get each suite's results into a single TAP output file for that suite with all of the test cases numbered from 1..n

However one of the suites I have to integrate consists of a number of .t files. Each one generates TAP. I've tried running all of these using TAP::Harness. This runs the tests fine, however the final output from:

runtests (@testfiles);

is no longer in TAP format.

When the framework completes I parse each of the final results TAP files from each test suite using TAP::Parser, to generate a single report for each component.

I could of course set STDOUT to a file and do something like this:

for each my $testfile (@testfiles){system ("perl $testfile"); }

But if I do that I'll obviously get TAP 1..n for each test file rather than 1..(total number of test cases in all files)

If this is clearer than mud, could anyone tell me whether there is a way to either:
a) run all test files and generate a single TAP output file 1..n test cases total OR
b) parse n TAP files where each one presents results 1..n for each test file into a single comprehensive output

I'm hoping not to have to do any editing to the already existing .t files in this test suite.

Thanks for the help,

Eric

Replies are listed 'Best First'.
Re: TAP question
by ysth (Canon) on Feb 26, 2008 at 05:19 UTC
    You could have an overall .t file that runs each other test file with TAP::Harness and outputs an ok or a not ok with diagnostic info for each existing .t file.

    But wouldn't it be better to allow for more than one .t file per suite? If you aren't already aware, there have been dramatic improvements in the flexibility of Test-Harness since perl 5.8.8 that may be of assistance to you.

    Much discussion of testing takes place on the perl-qa list. You may want to check it out if you don't find the answers you need here.

    --
    Online Fortune Cookie Search
[reply]

      I believe that s/he already has multiple .t files per suite, but wants a single TAP output from each set. Or, something to take the TAP output from each .t files, and turn it into a single TAP output.

      I'd take a look at TAP::Parser, which at least seems like it would help you roll your own solution if there wasn't a prebuilt module.

[reply]
        I believe that s/he already has multiple .t files per suite, but wants a single TAP output from each set. Or, something to take the TAP output from each .t files, and turn it into a single TAP output.
        Yes, I understood that much. But the OP wants single TAP output because "I'm building a test framework that tests a number of different components, each with it's own suite of test cases".

        It doesn't seem like it's going to be substantially easier to coerce multiple TAP streams into one and then feed that to the test framework than to just make the test framework support multiple tests per suite in the first place. Probably the contrary, in fact.

        --
        Online Fortune Cookie Search
[reply]
      Interesting idea. This could work. I can't make any mods to the existing .t files to make them provide a return value since I don't own them, but I could manually parse the output stream looking for "not ok", and if found, trigger a failure for the entire file. Thanks!
[reply]
        I wasn't suggesting manually parsing them. If you just look for "not ok", for instance, you lose the ability to have TODO tests. I was suggesting having a single .t file that uses TAP::Harness to run the existing .t files and give a per-file ok or not ok result (that your outer framework interprets).
        --
        Online Fortune Cookie Search
[reply]

Back to Seekers of Perl Wisdom