A test harness for scripts

When it comes to modules, there are well established techniques for writing tests. With a standard CPAN module, there are test scripts provided - either a single script test.pl, or a directory of scripts, t/*.t.

In most cases, the tests operate on one or more .pm modules, exercising their functions, method calls, etc. testing out the scenarios. Sometimes a CPAN module delivers one or more "exe_files", which are scripts. Examples of this are dbish and lwp_rget. I also count sub shell inside CPAN.pm and CPANPLUS.pm as a script: it's just a script that is wrapped up inside the .pm module itself.

Some people may argue that scripts that accompany modules do not require tests as they contain no functionality - merely calls to the .pm module code. I disagree.

As far as I can see, these scripts are delivered with little or no collateral by way of accompanying documenation and tests (though there are probably exceptions to this). As far as I am concerned, the "exe_files" are deliverables which also need tests to verify that they are working correctly. The nature of the script in question, and its tests, could be one of the following:

Command shell scripts. These take lines of input from stdin, and display results on stdout and stderr. I envisage the tests as using IPC::Open3 to fork a process to run the script, passing commands and checking returned text against expected results. The tests can also be examining external information, such as the file system or a database.
Daemon scripts. Besides considerations that the daemon needs to be isolated from any "live" instance of the daemon alteady running, the tests need to handle and check for start up and shut down of the daemon, together with passing messages to and from the daemon.
Batch scripts. These are probably the simplest, as it may be merely a matter of running them under system or backticks and checking expected results.
GUI scripts. These must be the hardest to test. For example, Tk provides access to the bindings, to allow automation of mouse and keyboard events. But it's still very difficult to automate.
Web scripts. I include CGI and mod_perl scripts here, along with client side LWP and IRC chatbots. What would be really nice is something that could fake a webserver, and generate the HTTP requests that the likes of CGI.pm is expecting.

As is apparent, I don't think that there is a single one-size-fits-all test harness for scripts, but I am interested in whether any modules exist which cover the framework of testing each of the types of script above. I'm also keen on keeping the tests portable, hence not relying on operating system dependent IPC mechanisms.

--
I'm Not Just Another Perl Hacker

Comment on A test harness for scripts Download Code

Replies are listed 'Best First'.
Re: A test harness for scripts by dragonchild (Archbishop) on Feb 07, 2004 at 16:54 UTC
I've had some very good success with the Test::Cmd distro, when testing commandline stuff. I used it to test something that was going to be run in a batch mode, ending up with over 200 tests in less than a month of ad-hoc development. This works equally well for commandshell and batch, cause batch is really just commandshell w/o an active user. As for web scripts, I've been getting some good traction using a combination of the following: Structure the application using CGI::Application and with good modularization. Standard 'make test' works for the API. (My current app is split into 9 CPAN-like distributions and one catch-all install including the templates, CSS, and C::A bootstrap scripts.) WWW::Mechanize, which not doing look-and-feel very easily (you have to source-dive the HTML) and not handling JScript events (you have to implement them yourself in the test script), it does do a good job of spot-checking the result. Plus, it nicely handles redirects and the like. Good luck! ------ We are the carpenters and bricklayers of the Information Age. Please remember that I'm crufty and crochety. All opinions are purely mine and all code is untested, unless otherwise specified.	[reply]
Re: Re: A test harness for scripts by petdance (Parson) on Feb 10, 2004 at 02:13 UTC
WWW::Mechanize, which not doing look-and-feel very easily (you have to source-dive the HTML) You need to discover the mech-dump tool that comes with WWW::Mechanize. You feed it a file on disk, or a URL, and it tells you all the forms and fields on it. $ mech-dump http://perlmonks.org GET http://perlmonks.org/index.pl node= go_button=Search (submit) POST http://perlmonks.org/index.pl node_id=131 (hidden) op=login (hidden) user= passwd= (password) expires=<UNDEF> (checkbox) [*<UNDEF>/off\|+10y/rememb +er me] login=Login (submit) POST http://perlmonks.org/index.pl node_id=131 (hidden) foo=Refresh (submit) POST http://perlmonks.org? node_id=325590 (hidden) displaytype=display (hidden) vote=<UNDEF> (radio) [0/TCP\|1/UDP\|2/ICMP\|3/IPX\| +4/SMB\|5/TFTP\|6/SMTP\|7/NNTP\|8/HTTP\|9/WAP\|10/SOAP] <NONAME>=Vote (submit) [download] xoxo, Andy	[reply] [d/l]
Re: A test harness for scripts by hardburn (Abbot) on Feb 07, 2004 at 14:18 UTC
Shell scripts could also be tested by using a `DEBUG` var to suppress their normal output and run a "test mode" that does all the normal Test::Harness output. This solution would probably get complex fast. IPC::Open3 will likely be better. Basic CGIs can be run on the command line by faking the HTTP request to the program's STDIN. You might be able to do the same with mod_perl code if it's not too fancy. I don't think there is a good way to test ApacheHandlers without an actual Apache server. If you use HTML::Template, and if you don't mind a blatent plug for my own module, you can use HTML::Template::Dumper to remove all the "design fluff" in the CGI output. It allows you to directly inspect the datastructure sent to the template. Using it in a test situation is exactly what it's designed for. ---- `: () { :\|:& };:` Note: All code is untested, unless otherwise stated	[reply] [d/l] [select]
Re: A test harness for scripts by drfrog (Deacon) on Feb 08, 2004 at 09:29 UTC
take a look at this Apache::Test its included in the modperl tarball as well as the bug report mechanisms... ...basically it starts up an apache process and can run and test mod_perl handlers, as well as cgi-bin/ scripts via various .t's. check the examples httpd.apache.org/test has other stuff i havent checked into in a while... the other one ive spent some time with is HTTP::WebTest which works pretty good as well there are others for testing various other things , such as seige for testing a server with different amounts of users... or the w3c html validator for linkages	[reply]
Re: A test harness for scripts by etcshadow (Priest) on Feb 08, 2004 at 20:42 UTC
Well, what I'd say more than anything else is: put as little logic as possible in your scripts. Make them just executable wrappers around your modules, and then write test cases for your modules. Beyond that, yes, it would be cool to have script harnesses (although, my limitted experience with IPC::Run would lead me to recomend it over IPC::Open3). Last, as for testing web servers: what we do where I work is we actually wrote a web replay agent, which reads in apache logs and issues requests to the web server, logs the output and diffs it against the expected results. We've been trying to push the idea to management of getting this test harnass released to open source... but no luck yet. Anyway, the idea is pretty straight-forward, though. `------------ :Wq Not an editor command: Wq` [download]	[reply] [d/l]
Re: A test harness for scripts by petdance (Parson) on Feb 10, 2004 at 02:17 UTC
You may be interested in how I do it in Test::Harness and WWW::Mechanize: http://search.cpan.org/src/PETDANCE/WWW-Mechanize-0.72/t/mech-dump.t and http://search.cpan.org/src/PETDANCE/Test-Harness-2.40/t/prove-switches.t This is also sort of being discussed over on perl-qa. If someone wants to write up a summary of what's available, please do it in POD (or HTML if you must) and mail it to me so that I can put it up on http://qa.perl.org. xoxo, Andy	[reply]