Test failures matter

szbalint has asked for the wisdom of the Perl Monks concerning the following question:

Dear monks,

I've recently released WWW::Curl 4.00 to CPAN.

I've waited a few days and noticed that most of my cpan test failures in the latest release are not definite test errors, but rather cases of :

Confused test output: test 9 answered after test 12
Confused test output: test 10 answered after test 16
Confused test output: test 11 answered after test 19
Confused test output: test 12 answered after test 20
[download]

This made me wonder, what's going on? The test file in question is singlethreaded and uses Test::More's 'ok', plus a for loop. How is it possible then, that test output gets printed in the wrong order?

I'm either missing something really obvious or this is something arcane.This test is responsible for most of the FAIL reports, however there is another interesting case.

You see, WWW::Curl requires libcurl to be installed, so I've included requires_external_bin 'curl-config'; in Makefile.PL to provide a rudimentary check for the existence of the library. Still, FAIL gets reported instead of NA that I intended if the prerequisite library isn't available. Since this is clearly not it, what is the proper way to have automated testing degrade gracefully to NA when an external build time dependency like a library isn't available and because of that there is no hope of building the library properly?

update: the latter turned out to be a bug in Module::Install, which should be fixed in the next M::I release.

update 2: the former was an interesting bug, WWW::Curl kept spewing out debugging/header output under some circumstances, which applied to ssl debugging aswell. That happened to be of course non-human readable binary output - without \n endings, so when Test::More tried to output "ok 12 - whatever", it got appended after that binary gunk. This caused Test::Harness to get confused.

Comment on Test failures matter Select or Download Code

Replies are listed 'Best First'.
Re: Test failures matter by dragonchild (Archbishop) on Apr 17, 2008 at 12:49 UTC
Your test is single-threaded. Is curl? My criteria for good software: Does it work? Can someone else come in, make a change, and be reasonably certain no bugs were introduced?	[reply]
Re^2: Test failures matter by szbalint (Friar) on Apr 17, 2008 at 13:02 UTC
As far as I know, yes.	[reply]
Re^3: Test failures matter by andreas1234567 (Vicar) on Apr 17, 2008 at 18:49 UTC
Multi-threading: libcurl is completely thread safe, except for two issues: signals and SSL/TLS handlers. Maybe that's why the ssl tests come in wrong order? -- When you earnestly believe you can compensate for a lack of skill by doubling your efforts, there's no end to what you can't do. [1]	[reply]
Re^4: Test failures matter by szbalint (Friar) on Apr 17, 2008 at 19:44 UTC
Re: Test failures matter by Anonymous Monk on Apr 17, 2008 at 12:57 UTC
Alien?	[reply]
Re^2: Test failures matter by szbalint (Friar) on Apr 17, 2008 at 13:19 UTC
I wouldn't like to repackage libcurl into Alien:: (distributions should be able to handle their own dependencies), just have a way to clearly signal to automated and interactive users alike that WWW::Curl depends on a library.	[reply]
Re: Test failures matter by Anonymous Monk on Apr 17, 2008 at 13:04 UTC
this fail is a bug in CPANPLUS, it should report NA.	[reply]
Re^2: Test failures matter by bingos (Vicar) on Apr 17, 2008 at 17:33 UTC
Which is a shame, because this test report was generated by CPAN::Reporter not CPANPLUS.	[reply]
Re^3: Test failures matter by Anonymous Monk on Apr 18, 2008 at 10:34 UTC
Same difference, PERL5_CPANPLUS_IS_RUNNING = 1884	[reply]
Re^2: Test failures matter by szbalint (Friar) on Apr 18, 2008 at 13:20 UTC
It was actually a bug in Module::Install, which should be fixed in the next release.	[reply]
Re: Test failures matter by Anonymous Monk on Apr 17, 2008 at 13:07 UTC
Probably a bug in Test::More	[reply]