RFC: Test::Cases -- Worth releasing?

xdg has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks,

I'm finding a lot of my testing boils down to taking some input, manipulating it, and comparing it to some output. It's fairly simple to put the inputs and expected outputs in a data structure and loop over them (c.f. my coment in Re^2: Wanted, more simple tutorials on testing). However, as I add cases, the data structure winds up dominating the test file and managing it or finding things in the middle when a case fails become challenging.

One option to clean it up a bit is putting cases into the __DATA__section and pulling them out from there. This is the approach that Test::Base takes. Test::Base is really cool, but I'd rather avoid that because it's a Spiffy module and I wouldn't want that as a dependency just to test something I wrote that isn't using Spiffy already.

What I've been doing lately instead is sticking my test cases into individual files in a subdirectory, and then iterating over them as so:

For each file:

Slurp the file
Split it on a separator string into "input" and "expected"
Process the input
Compare processed input to expected output with a Test::More function like is(), using the name of the file as the test label

Rather than repeating that in every test file, I've put it into a test helper module that also:

Does sanity checks that a target directory exists, is readable, etc.
Searches the directory recursively to assemble a sorted list of files
Sets the plan to the number of files if a plan hasn't already been defined
Takes a callback for the input processing function -- e.g. transforming it directly or using it as a parameter to some other function

It would be fairly trivial to package this up as a module for CPAN and add some additional callbacks so that it manages the file manipulation and users only have to provide processing functions and a test function if the default is() test isn't what's needed.

It's not really doing all that much -- but it does avoid having to write the same helper code over and over for common input/output testing and makes managing test cases as simple as manipulating small text files in a directory tree. Is there demand for this? Would people find it useful enough to include as a dependency with their own modules? (Have I overlooked some existing test module that does this already?)

For a name, I'm considering just Test::Cases or maybe Test::Casefiles. Other ideas?

Thanks,

-xdg

Code written by xdg and posted on PerlMonks is public domain. It is provided as is with no warranties, express or implied, of any kind. Posted code may not have been tested. Use of posted code is at your own risk.

Comment on RFC: Test::Cases -- Worth releasing? Select or Download Code

Replies are listed 'Best First'.
Re: RFC: Test::Cases -- Worth releasing? by sgifford (Prior) on Sep 16, 2005 at 16:10 UTC
I like to keep my input and output in two seperate files, say `test1.test` and `test1.ok`. My test scripts put their output into `test1.test`, and I use `cmp` from a shell script to compare the test output to the correct output. What's nice about this is if my test fails, I can run `diff test1.{test,ok}` to see what changed that made it fail, or a whole variety of other shell or perl tools designed for showing the differences between files. And if I find that the new test output is good, I can just type `cp test1.{test,ok}` to make it the new correct output.	[reply]
Re^2: RFC: Test::Cases -- Worth releasing? by xdg (Monsignor) on Sep 16, 2005 at 16:39 UTC
That's an interesting approach. I've tried similar things at times but usually found that maintaining the separate files meant more work keeping my inputs and outputs in sync -- but I could see that approach being better if the files themselves are particularly long. As for `diff`, I actually use Test::Differences instead of Test::More's `is()` (but only if it's available so end users won't need Test::Differences as a dependency) to give me that kind of comparison, but I could see that direct access to `diff` would make some complex cases easier to figure out. (I wish I could get unified context diffs from Test::Differences.) -xdg Code written by xdg and posted on PerlMonks is public domain. It is provided as is with no warranties, express or implied, of any kind. Posted code may not have been tested. Use of posted code is at your own risk.	[reply] [d/l] [select]
Re^3: RFC: Test::Cases -- Worth releasing? by sgifford (Prior) on Sep 16, 2005 at 16:49 UTC
Yeah, I tend to use this for test that have a few hundred lines of output, which are useful for testing the final results of a system. For example, I often generate complete HTML pages to make sure everything in a Web application has worked properly. One trick that's nice is if I know what's changed, I can see if there are any differences besides that with some shell cleverness. For example, if I've added an output field called `some_thing` to my test file, and the only thing that should have changed is the addition of that one field, I can run something like: `diff -u <(grep -v some_thing test1.test) test1.ok` [download]	[reply] [d/l]
Re: RFC: Test::Cases -- Worth releasing? by skillet-thief (Friar) on Sep 18, 2005 at 10:46 UTC
My comments here should be taken with a fairly healthy grain of salt, since I am more of a beginning tester than a power tester, and since my experience is mostly limited to Test::More. I definitely don't know all of what is out there. That said, a module like `Test::Cases` (`Test::Casefiles` seems a little clearer to me) would have been useful for me (or might still be) when I was doing a lot of html filtering and parsing. Mainly as a turnkey setup for dealing with all those files. Putting input and output in the same file seems like a pretty clear approach. This might have been the kind of module that I would have been glad to stumble on while wandering CPAN. So there are my 2cts. `sub jf { print substr($_[0], -1); jf( substr($_[0], 0, length($_[0])-1)) if length $_[0] > 1; } jf('gro.alubaf@yehaf');` [download]	[reply] [d/l] [select]
Re: RFC: Test::Cases -- Worth releasing? by qq (Hermit) on Sep 19, 2005 at 21:00 UTC
Release it. I just wrote very similar code for a bunch of xml filter modules. Take some xml, change it, spit it out. I use a data directory with pairs of source/expected files (They get named `test_name.a.source, test_name.a.expected, test_name.b.source, test_name.b.expected,` ...) If there is failure, I write a failure file to the same directory, with a timestamp appended. This is for easy diffs. I'd definitely like to have separate source/expected files. I still have individual `.t` files, so I can add tests that don't fit into the other structure.	[reply] [d/l] [select]
Re: RFC: Test::Cases -- Worth releasing? by InfiniteSilence (Curate) on Sep 16, 2005 at 15:10 UTC
Is this a question, or should this be in Meditations? Celebrate Intellectual Diversity	[reply]
Re^2: RFC: Test::Cases -- Worth releasing? by xdg (Monsignor) on Sep 16, 2005 at 15:28 UTC
Considering the title and the number of question marks in the last couple paragraphs, I'd say it's a question. Is it a question about how to do something in Perl -- or, more typically for SoPW, please debug my regex, help me find my variable typo from lack of `strict`, explain HoH data structures, etc. -- well, no. I debated posting it as a "Meditation" -- but it didn't seem particuarly enlightening or much of an epiphany. (See Where should I post X?). But maybe it's an epiphany for some. Ohhhhmmmmm.... -xdg Code written by xdg and posted on PerlMonks is public domain. It is provided as is with no warranties, express or implied, of any kind. Posted code may not have been tested. Use of posted code is at your own risk.	[reply] [d/l]