Test coverage and canned dog food.

Squeeks has asked for the wisdom of the Perl Monks concerning the following question:

For my first significant (non Acme::) distribution for CPAN, I've been able to create modules that will interact with a propriety database and in trying to create the tests for all the interactions, have hit a stumbling block - to supply canned data, or not to.

Normally I would ask WWDBID (What Would DBI Do?), however RDBMS-like databases allow permitted users connecting to the server to not just modify rows, but also structure as well. This is a freedom unavailable to me given the way the software works. In conclusion I see a few options:

Can it - supply a small canned data set so non-smoke testers and end users with the propriety software can test it. This means putting a 100kb+ binary blob into the distribution, and also into my repository.
Linked can - don't supply the canned data, but simply host it, so that interested parties can download it and use it as they wish. In the future when I might care less about developing this, this link could break.
Hijack - use an existing database hosted on the client's test server and perform both read and write queries. This has the potential to go wrong if bad code deletes or overwrites end user data.
Mouse Potato - Don't bother about complete test coverage and go back to browsing tech blogs.

Wise monks, what do you suggest?

Comment on Test coverage and canned dog food.

Replies are listed 'Best First'.
Re: Test coverage and canned dog food. by almut (Canon) on Mar 03, 2010 at 10:45 UTC
Personally, I would favor "putting a 100kb+ binary blob into the distribution", because it has the highest likelihood to actually work for an extended period of time, and having to download in the order of 100k more isn't such a big issue these days.	[reply]
Re^2: Test coverage and canned dog food. by DrHyde (Prior) on Mar 03, 2010 at 11:36 UTC
Agreed. And you wouldn't be the first to do this either. eg, take a look at http://cpansearch.perl.org/src/EXIFTOOL/Image-ExifTool-8.10/t/images/ or http://cpansearch.perl.org/src/JV/eps2png-2.7/t/.	[reply]
Re: Test coverage and canned dog food. by JavaFan (Canon) on Mar 03, 2010 at 14:00 UTC
however RDBMS-like databases allow permitted users connecting to the server to not just modify rows, but also structure as well. Eh, no, unless you give out grants that way. RDBMS typically have a fine-grained permission system, defining who can see or modify what. Wise monks, what do you suggest? I don't understand the options. I can see a choice between "Can it" and "Linked can", but "Hijack" seems to be a different decisions. The decision to "Hijack" or not seems independent to "Can it" or "Linked can". As for the Hijack choice, I suggest using "Test::Database". If the installer wants to run test, (s)he'll have to install "Test::Database", which makes him/her configure a test database for use. This will give your tests a safe sandbox to play it. If Test::Database isn't installed, or not configured to be used with the database you require, you skip the tests.	[reply]
Re: Test coverage and canned dog food. by MidLifeXis (Monsignor) on Mar 03, 2010 at 14:17 UTC
How about canning it, but with something from the Mock series of modules? That way you don't even need to have to require that your testers have the proprietary modules. Of course, it is also possible that I am missing what you are actually trying to test. It is said that "only perl can parse Perl." I don't even come close until my 3rd cup of coffee. --MidLifeXis	[reply]
Re: Test coverage and canned dog food. by CountZero (Bishop) on Mar 03, 2010 at 16:34 UTC
Why not do it all? If the propriety software is available and the grants are OK: use the canned data in the distibution on that software; if not fall back to Test::Database and use the canned data there; if Test::Database is not available, fall back to one of the Mock modules. CountZero A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James	[reply]