I have been pondering a problem faced by my coworkers and wondering if a task that seems huge and difficult might be made easier with Perl.
Many of the projects I support at work involve tremendous amounts of code written in multiple languages, some OO, some not. Almost all of these projects are either required, or strongly desire, to prove that their test cases cover the paths of execution of the code.
Unfortunately, most commercial tools we've been able to find only provide a GUI or script based front-end for the user to build test cases, then the tool will run the test and report the code coverage. This process is highly manual, and I don't think it takes full advantage of the abilities of the computer.
What I would like to begin working on is a more advanced front end that would:
I do NOT want to remove user scrutiny from the testing process, put I would like to help the tester get a better running start.
I would like to hear any thoughts about the possibility of such a system, any problems you see as interesting, and most importantly, what tools, modules, and data formats you would use if you were trying to build such a thing.
|
Re: Automated Path Coverage Test Case Generation
by Ovid (Cardinal) on Oct 03, 2003 at 16:05 UTC
|
One way to approach this would be to ascertain what coverage you currently have and then keep writing tests until you have satisfactory coverage. Devel::Cover can help with this. For example, here are the first few lines of a table that lists the sort of coverage you would get:
I've left in the hyperlinks, but I changed them so that they don't actually go anywhere. Were you to actually generate such a report, those hyperlinks would go down into line-by-line detail pages that show you exactly what is and is not getting covered. You'll even find truth tables that show you which conditions of a logical expression are getting used!
By reading through Devel::Cover::Tutorial, you can learn what those headings mean. Generating such a report is easy. Here's how I did it:
perl -MDevel::Cover /usr/local/bin/web_test -d -h
# wait a while for the tests to finish
cover cover_db -report html
I number at the far right is a loose estimate of the percentage of code coverage. It's also fair to note that there are a few bugs in this code (it's alpha code, after all). That report came up fine the first time that I ran it. The second time, I found that the index file had been named .html, making it invisible with a standard ls. I had to rename it to index.html and copy it to the cover_db/ directory to get it to work properly.
Despite a few bugs, it's very easy to work with and can give you a nice idea of how complete your coverage is. Since the html is generated through Template Toolkit, you might even be able to auto-generate a few tests with this ...
| [reply]
[d/l] |
|
|
| [reply] |
|
|
Yes, this is a "Perl only" solution. It's intimately tied to the internals of Perl. I don't know what's available for the other languages. If you find out, posting the answer here would be great (though probably as a reply to this thread since that would be such an OT root node that some monks might not like that :)
| [reply] |
Re: Automated Path Coverage Test Case Generation
by Corion (Patriarch) on Oct 03, 2003 at 14:43 UTC
|
I can imagine two ways for this, one easy and one hard:
The easy way would be to write a rudimentary parser that recognizes subroutines, be they methods or plain functions, and within those, recognizes loops and conditionals. For every loop and conditional, it generates mock up code that has possibly to be modified by a human to exercize the boundaries/alternatives.
The hard way is to have a parser for the language to find out what the procedures/methods and conditionals and loops are. Then you can also apply some heuristics to generate from a "known good" test case the negated test case to test for some opposite.
But I would go with the simple approach of automatically generating skeleton tests that reference back to the original code and have these manually fixed by a human.
perl -MHTTP::Daemon -MHTTP::Response -MLWP::Simple -e ' ; # The
$d = new HTTP::Daemon and fork and getprint $d->url and exit;#spider
($c = $d->accept())->get_request(); $c->send_response( new #in the
HTTP::Response(200,$_,$_,qq(Just another Perl hacker\n))); ' # web
| [reply]
[d/l] |
|
|
If I were to try the easy route, should I use some of the Template modules to build the mock-ups?
The hard way would be the more desirable result, but much more complicated.
Thanks for the ideas.
| [reply] |
|
|
Using any templating system is an idea that escaped me until you mentioned it, but yes, using a (simple) templating system like HTML::Template is a good idea, as it allows you to also easily output "relevant" metadata without having to change your code.
Another bit halfway between the easy and the hard solution is to use Acme::Test (respectively a close relative of it) to generate the skeleton tests, as it already does the gruntwork necessary for the upper level of the test skeleton (would that be the skin?).
If you combine that approach with a templating system, you can already have the templates for loop and conditionals testing in place, even if your parser does not yet produce the data structures for these.
Another thing is that the templating systems also allow for easy generation of the multiple permutations for loops and conditionals. One thing that might be a downside of HTML::Template is, that it dosen't allow for one template going out into several files, but you can still split up the string generated by it into several strings and use them as separate files.
perl -MHTTP::Daemon -MHTTP::Response -MLWP::Simple -e ' ; # The
$d = new HTTP::Daemon and fork and getprint $d->url and exit;#spider
($c = $d->accept())->get_request(); $c->send_response( new #in the
HTTP::Response(200,$_,$_,qq(Just another Perl hacker\n))); ' # web
| [reply]
[d/l] |
|
|
