ISO fixtures for testing internet client code

tlm has asked for the wisdom of the Perl Monks concerning the following question:

For testing internet client code it would be very useful if there were a canonical server that served as broad a range of conditions (particularly anomalous ones) as possible. (Hey, a monk can dream.) I am familiar with http://www.example.com/index.html (and its siblings), but this is useful for the most basic, and least problematic, of HTTP communication.

A second best would be some ready-made package of files and accompanying howto for setting up such a server locally (ideally it would be possible to distribute this "test server package" along with one's module, so that users of the module had the option of adding the capability of testing more communication conditions using their own servers).

There's also the "Potemkin solution": use something like Test::MockObject to create a false interface to, say, LWP::UserAgent, and have it generate mock user agent objects returning mock HTTP response objects, for example. This alternative is least desirable because it could only produce a test suite that was as good as the author's grasp of the protocol(s) involved, but, arguably, such grasp is, at bottom, one of the things an ideal test suite would test (assuming that the code and the tests are authored by the same person, as is often the case).

My question to you, fellow monks, is what "fixture" servers do you use when you want to test internet client code? In particular, how do you ensure adequate coverage of those parts of your code designed to check for and handle anomalies in the communication? How do you make these tests portable?

the lowliest monk

Comment on ISO fixtures for testing internet client code

Replies are listed 'Best First'.
Re: ISO fixtures for testing internet client code by Corion (Patriarch) on Jun 04, 2005 at 18:08 UTC
While writing the tests for WWW::Mechanize::Shell, I wrote me Test::HTTP::LocalServer, which spawns a local (single-threaded) HTTP server which logs and replies in a very predetermined manner when I request special URLs.	[reply]
Re: ISO fixtures for testing internet client code by nothingmuch (Priest) on Jun 04, 2005 at 22:32 UTC
That depends on what you want the tests to cover. You implied that you use LWP in there, so I assume your layout is layered. Let me make a guess of what an 'internet client' might look like: LWP resource retrieval code processing of result, oblivious to fetching composition of fetching/processing output In my opinion if you're going to test the software, you ought to test that each part works as expected, and also test that they work toegether as a whole - unit testing and integration testing. If you're at the unit testing stage I suggest you ignore complicated webserver setups, and simply use Test::MockObject to make sure that each layer behaves properly based on some precanned output from the mocked layer underneath it. If you're doing a larger scale test for the way the things behave together, then it's a question of what's easier, and breaks less. If you can easily mock server side errors, connection drops, edge-case data, and all the other stuff testing should help you make sure you cover by mocking a webserver, by all means go for it. There are many ways to write a simple HTTP server on CPAN. If it looks like it will be simple to mock LWP, on the other hand, go with that. There's no cost quality wise, since LWP is not your code - it's tested anyway. This has the advantage of being more readable and simpler, since there are no multiple processes or invocations involved. It might even be easier to generate truely odd cases, which is valuable for test cases based on bug reports. The cost is that you don't really know whether your layers use LWP properly, since "real" LWP is not involved. I don't think this is the same as your rationalization for why it's the least desirable - 100% coverage is often too much. The effort spent getting from 90 to 100 is probably going to be better spent on alternative methods of quality control. I would say that my best advice is to try and balance a throwaway HTTP server with mock objects. As for example.com and friends - this might introduce false positives into your test suite, if the net is down, for example. If you do go ahead with it, make sure to query the user before initiating a connection though (A good place for this is Makefile.PL). -nuffin zz zZ Z Z #!perl	[reply]
Re: ISO fixtures for testing internet client code by mpeters (Chaplain) on Jun 04, 2005 at 23:40 UTC
Apache::Test, LWP::UserAgent it's child WWW::Mechanize, and it's grand-child Test::WWW::Mechanize are all I've ever needed. Apache::Test lets you embed a working Apache, with or without mod_perl (and even with PHP if you're into that sorta thing) with your application/module and you can then use the LWP based modules to actually perform the HTTP requests. More people are killed every year by pigs than by sharks, which shows you how good we are at evaluating risk. -- Bruce Schneier	[reply]