Counting test cases

Often have I seen or written a line like this:

use Test::More 'tests' => 23;
[download]

This tells Test::More that there will be 23 tests. If I add or remove a test, I also have to update the number of tests. I usually don't have to count them myself, though. I can just run the test script and see how many tests it runs. Nevertheless, this small extra hassle is one I'd like to avoid.

Sometimes I have data driven tests, and it looks like this:

use Test::More;

# test definitions

plan 'tests' =>       scalar @cases
                + 2 * scalar @cases_that_cause_two_tests
                + 17;
[download]

In this example, I have 17 tests that run regardless of the data, and I have some other tests based on data structures. Now when I make a change, it's not as easy to find out what that number should be. Running the script gives me some number > 17, but how many of those were driven by data and how many were the static tests?

So I've devised another way to count tests.

use Test::More;

# This variable is set by BEGIN blocks elsewhere.
my $static_tests;

plan 'tests' => $static_tests;

BEGIN { $static_tests++ }
pass 'Static test 1';

BEGIN { $static_tests += 3 }
pass 'Group test 1/3';
pass 'Group test 2/3';
pass 'Group test 3/3';
[download]

I think it's important to document the $static_tests variable because it's not immediately obvious where it gets its value.

In a real test script, I might want to stagger the counters. I could have a single BEGIN { $static_tests += 10 } for every ten cases. A longer test with a lot of setup would have a BEGIN { $static_tests++ } with it. I don't have to have more book keeping than actual testing.

I wonder if remembering the BEGIN block to go with the tests will become a problem. As long as the script isn't too long, it shouldn't be hard to go through it and make sure they all match up as they should. If the script is too long for that, I probably should break it up anyway.

I'd be interested to hear the thoughts of the other monks about this.

Comment on Counting test cases Select or Download Code

Replies are listed 'Best First'.
Re: Counting test cases by arc_of_descent (Hermit) on Mar 29, 2008 at 06:04 UTC
You might consider using Test::Class which is great for organizing your test suite into libraries. In your test module, you then define a subroutine which runs the tests and pass it the number of tests. That way, you don't really need to know the total number of tests, just the number of tests in a subroutine which ideally would test a particular feature or portion of your code base. `package MyTestModule; use base qw/ Test::Class /; use Test::More; sub test_sub1 : Test(2) { my $self = shift; ok(....); ok(....) }` [download] When you update the test_sub1 subroutine to include or remove a test, just change the number in the attribute to the sub. Very convenient! You can also do this. `sub test_sub2 : Test(no_plan) { my $self = shift; ... ... }` [download] -- Rohan	[reply] [d/l] [select]
Re: Counting test cases by Herkum (Parson) on Mar 29, 2008 at 14:22 UTC
It is something that Test::Class already handles in the way it divides up tests, but I think it is an interesting approach that I have never thought of. I love seeing ideas like this. Even though I may not use it , I get to see new ways to make code work.	[reply]
Re: Counting test cases by jplindstrom (Monsignor) on Mar 29, 2008 at 17:59 UTC
Test::Class is ideal for this, and in addition it makes test fixture setup a lot more manageable. As such it is more suitable than Test::More for larger code bases, like entire applications (as opposed to library/framework style CPAN modules). Even for smaller modules, it provides a good balance between supporting structure and ease-of-use, so I'd look into that if I were you. But if you can't solve the problem, there's always the option to automate it. /J	[reply]
Re: Counting test cases by talexb (Chancellor) on Apr 01, 2008 at 15:47 UTC
Interesting post. The problem I'm trying (not very hard) to solve is a non-fatal test that just tries all of the links on a given page. I don't know how many links there are, and I don't really care -- I just want to find out which links work and which links fail. However, it seems that Test::More wants to know at compile time how many tests there are, and I only know how many links there are to test at run time. Alex / talexb / Toronto "Groklaw is the open-source mentality applied to legal research" ~ Linus Torvalds	[reply]
Re^2: Counting test cases by kyle (Abbot) on Apr 01, 2008 at 16:02 UTC
The solution is hiding in the OP. I'll spell it out... `use Test::More; # note I don't say how many there are # do what you have to do to get your list of links my @links = list_of_links(); # this happens at run time plan 'tests' => scalar @links; foreach my $link ( @links ) { ok( link_success( $link ), "link '$link' is good" ); }` [download] You can call `Test::More::plan` any time before you do the first test, and you can do as much test setup as you want before that.	[reply] [d/l] [select]
Re: Counting test cases by szabgab (Priest) on Mar 30, 2008 at 17:41 UTC
I usually group together blocks of tests: `{ BEGIN { $static_tests += 3 } pass 'Group test 1/3'; pass 'Group test 2/3'; pass 'Group test 3/3'; }` [download] It has several advantages, one of them is that I have to count only a small number of tests. Another advantage is that now you can use the same variable names in each block without the risk of interference between them.	[reply] [d/l]
Re^2: Counting test cases by kyle (Abbot) on Mar 30, 2008 at 18:28 UTC
I do that too when I'm testing. I didn't do that in my OP because it didn't seem relevant to the topic. I actually put the counter block outside the group, though. `BEGIN { $static_tests += 3 } { pass 'Group test 1/3'; pass 'Group test 2/3'; pass 'Group test 3/3'; }` [download] I'm not sure if I like that better, but the reasoning behind it was that the block is operating on a variable outside the group, so it should be outside the group. If it's in the group, it might look like it's less relevant to the stuff going on outside, which is not the case.	[reply] [d/l]