I agree completely with others in this thread who have suggested that you should break your application into components that can be more easily tested separately from the rest of the app.

When I develop CGI apps, or mod_perl handlers, I usually use a combination of Test::More and Apache::test (included with mod_perl), along with a set of other testing modules like Test::Manifest, Test::Distribution and Test::Prereq.

Apache::test allows me to start up my own private Apache web server, listening on a non-standard port, where I can perform LWP requests against it and check for expected output. As you've already mentioned this technique definately has some limitations, such as if the HTML changes, then the tests break. I try to get around this problem by doing the following:

1. All code is version controlled. I use CVS mainly, but I may be moving to Subversion shortly.
2. All new code is bundled in small packages that are interrelated, and all the support files in one place (HTML templates, test files, installation scripts). I've modelled this after the way CPAN modules are bundled, and it works amazingly well.
3. Things like template names, database connection information, etc are pulled from external configuration sources. This might be an httpd.conf if I'm writing a mod_perl handler, or it could be a flat file/database if I'm writing a CGI.

The theme here is that I'm trying to make the testing environment as controllable as possible and reducing unknown factors.

I use version control so I can tag a group of files as being "stable", which means "if all of these files are checked out with this tag, the test cases should pass".

Keeping everything in self contained CPAN-style bundles and using Makefile.PL's gives me everything I need to check dependancies, run the unit test cases, and install the applications. I'd recommend looking into Module::Build if you're writing new code as a replacement for Makefile.PL and ExtUtils::MakeMaker -- I've been meaning to switch to it for some time, but haven't yet had a chance to learn its ins and outs.

By putting the template paths in external configuration, I can tell the application to use my simple pre-formatted HTML page, with all layout elements removed. The important thing is that I control the layout of this page. This allows me to write LWP scripts that can work with the application, parse its output, and check this against expected output. Side Note: depending on your application you may even be able to swap in DBD::CSV for your native database, thereby removing the application's dependancy on an external DB - I've only done this for smaller-ish applications, so some experimentation is probably necessary.

The downsides? Apache::test can be tricky to get working, but once you do it can be pretty easy to duplicate later on. The first few times you do this it will feel like you've taken on alot of extra overhead, all I can say It will pass with a few repetitions. Another thing is that this may be overkill for some applications, you'll have to decide if its appropriate or not. And finally you might be tempted to skip over conventional unit testing because you think this is enough. Don't, its not. This approach is complementary to doing unit testing, not a replacement for it.

Dan Kubb, Perl Programmer