Test Failure Best Practices, or when is it ok to "force install"

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks. I tried searching for this kind of question before posting, but had no luck, I apologize if it's been asked and answered.

I have a question regarding installing stuff from CPAN. Recently I was installing HTML::TableExtract, and I had a test failure. The test failure was something along the lines of expected "<some><html>" got "<some><slightly><different><html>". This led me to believe that it was a small bug in the code, a small bug in the test, or someone had changed their website. Now if it's the first two, that means that the author of HTML::TableExtract released it knowing their was a bug (Since I'm assuming he runs his own tests). This leads me to believe that whatever the bug was, it wasn't that big of a deal. Likewise if someone changed a website it's not really HTML::TableExtract's fault.

Even so this is not the first time I've had one or two tests fail while the package itself seems to operate fine after it's been installed. I'm wondering if their is a heuristic for deciding when it's "ok" if a test failed. Is a failed test always a sign of some misconfiguration on my end? Or do packages get released with known test failures? Is the fact that their was a release regardless of a failed test assurance enough that the bug "isn't that bad"?

I realize the ultimate answer to all these questions is to read the source of the package in question, but I don't always have the time or the understanding to do that. I thank you for your time, and look forward to your responses.

Comment on Test Failure Best Practices, or when is it ok to "force install"

Replies are listed 'Best First'.
Re: Test Failure Best Practices, or when is it ok to "force install" by rinceWind (Monsignor) on Jun 26, 2006 at 16:17 UTC
Test failures are an indicator of either a bug or a problem with your system. In most cases, the latter should also be classified as a bug, unless the module author has documented that it should not, or is not intended to work on platform X. As a first port of call, check the README or INSTALL. Just to make sure that this is a normal CPAN install. Secondly, check CPAN testers to see what install problems, if any, have been encountered by the smoke testers. This is linked from the search.cpan.org module page. Finally, check RT (also linked from the same place) to see if anyone else has reported this problem. If this is not the case, report the problem yourself to RT, which will result in the module author getting an email, and you have a ticket number that you can use to track progress. As to whether to proceed installing a module with failing tests, "Caveat updator"; on your conscience be it. If you can work out what the tests/module _should_ be doing, stick a patch into the RT ticket; the author will in most cases be grateful. -- Oh Lord, won’t you burn me a Knoppix CD ? My friends all rate Windows, I must disagree. Your powers of persuasion will set them all free, So oh Lord, won’t you burn me a Knoppix CD ? (Missquoting Janis Joplin)	[reply]
Re^2: Test Failure Best Practices, or when is it ok to "force install" by exussum0 (Vicar) on Jun 26, 2006 at 20:25 UTC
Test failures are an indicator of either a bug or a problem with your system. Or a bug in the test.	[reply]
Re^3: Test Failure Best Practices, or when is it ok to "force install" by rinceWind (Monsignor) on Jun 27, 2006 at 07:01 UTC
Indeed. A bug in a test is still a bug, and should be reported. -- Oh Lord, won’t you burn me a Knoppix CD ? My friends all rate Windows, I must disagree. Your powers of persuasion will set them all free, So oh Lord, won’t you burn me a Knoppix CD ? (Missquoting Janis Joplin)	[reply]
Re: Test Failure Best Practices, or when is it ok to "force install" by eric256 (Parson) on Jun 26, 2006 at 15:54 UTC
Sometimes the failure may be cause by a difference in the system that the module is designed on versus where you are running it. I.e. it was designed on a Linux machine, but you are installing it on a Windows machine. In this case failed tests could indicate bugs that the author didn't know about. It is always a good practice to check the bug list for that module and see if this is a known issue. ___________ Eric Hodges	[reply]
Re^2: Test Failure Best Practices, or when is it ok to "force install" by hsmyers (Canon) on Jun 26, 2006 at 16:11 UTC
All true--- In addition let the author know about same with example as needed. --hsm "Never try to teach a pig to sing...it wastes your time and it annoys the pig."	[reply]
Re: Test Failure Best Practices, or when is it ok to "force install" by albert (Monk) on Jun 27, 2006 at 09:02 UTC
When I get a failing test, I always figure out the reason for failure before doing a forced install. Even when there is only one failure of hundreds of tests, it can indicate a deeper issue, which I don't want some small problem to come back and bite me in the future. It is much easier to debug things when one has the indicator of a failure. Then, once I've figured out the reason for the problem, I'll report it to the author. While any module with net-based tests often have failures due to changing test data, it still is important to understand why tests fail, as there can be no simple algorithm to say things are OK. -a	[reply]
Re: Test Failure Best Practices, or when is it ok to "force install" by Khen1950fx (Canon) on Jun 27, 2006 at 06:13 UTC
Test failures are to be expected. Ideally, we would like to be able to install a module with no problems. That hasn't been my experience. I'd say %50 of the modules that I have installed have been problematic. The suggestions in the other replies are great--Read the Readme, Install, etc. pods. Debug, rerun the tests, but, for me, I always go back to the Makefile.PL and tinker with it until it looks right to me. Then install the module with make -ik, make -ikt; make test -ik, make test -ikt; make install -ik, make install -ikt. Sometimes I rerun make as: "make -B", then start the procedure all over again. I wish there was an easier way(smile)...	[reply]
Re^2: Test Failure Best Practices, or when is it ok to "force install" by adrianh (Chancellor) on Jun 27, 2006 at 09:36 UTC
Test failures are to be expected That's not been my experience - on a wide variety of systems (with the exception of Windows which I rarely use so can't really comment on). However I have wasted huge amounts of time debugging problems on other peoples systems where modules have been installed with failing tests because, surprisingly enough, the failing tests usually demonstrate that the module isn't working properly. Please. For the sanity of the maintainers. Don't install modules with failing tests unless you understand why they're failing :-)	[reply]