I'm trying to wrap my head around how I can test that my Test::MonitorSites module is appropriately handling url's which should fail their tests, without having that interpretted by the Test Harness as an installation stopping failure of the test suite.

I'd like for this to be able to support an automated install from cpan. But it seems that my test suite should test url's which are expected to fail. See tests #36 and #37 below which pass when they test the output from test #19 and #20 which failed.

I have a test script which exercises a method called ->test_sites(), which works through a hash built by the Config::Simple method off of an .ini file. For each site in the configuration file, it runs the tests requested, storing its test results for analysis and action (emailing logs, sms messaging critical failures).

The Test Harness which handles prove t/*.t, counts not only the tests generated by my t/12_exercise_test_sites_method.t script, but also the tests conducted by the module which it calls. If I run from a command line: $ perl t/12_exercise_test_sites_method.t, my test output looks like this:

1..45
ok 1 - Test::MonitorSites->can('new')
ok 2 - Test::MonitorSites->can('test_sites')
ok 3 - Test::MonitorSites->can('email')
ok 4 - Test::MonitorSites->can('sms')
ok 5 - Test::MonitorSites->can('_test_tests')

   .   .   .   

ok 15 - Config::Simple->can('param')
ok 16 - Config::Simple->can('vars')
ok 31 - The result returned a sites value
ok 32 - The sites value is an array
ok 33 - It includes the right number of sites
ok 34 - Successfully linked to http://www.perlmonks.com.
ok 35 -   .  .  .  and found expected content for http://www.perlmonks
+.com.
ok 36 - Not able to find non-existent site: http://www.example.com.
ok 37 -   .  .  .  and did not find expected content for non-existent 
+site: http://www.example.com.
ok 38 - Successfully linked to http://validator.w3.org/.
ok 39 -   .  .  .  and found expected content for http://validator.w3.
+org/.
ok 40 -   .  .  .  checked all links on this page
ok 41 -   .  .  .  and the validity of the html code was tested
ok 44 - Successfully linked to http://www.campaignfoundations.com.
ok 45 -   .  .  .  and found expected content for http://www.campaignf
+oundations.com.
# Looks like you failed 2 tests of 45.
[download]

Note that tests 17-30 are missing from this output. But see below.

The module being tested here redirects its test results to: /tmp/test_sites_output_ok, which reads:

ok 17 - Successfully linked to http://www.perlmonks.com.
ok 18 -  . . . and found expected content at http://www.perlmonks.com
not ok 19 - Successfully linked to http://www.example.com.
not ok 20 -  . . . and found expected content at http://www.example.co
+m
ok 21 - Successfully linked to http://validator.w3.org/.
ok 22 -  . . . and found expected content at http://validator.w3.org/
ok 23 -  . . . linked to http://validator.w3.org/
ok 24 -  . . . successfully checked all links for http://validator.w3.
+org/
ok 25 -  . . . linked to http://validator.w3.org/
ok 26 -  . . . html content is valid for http://validator.w3.org/
ok 27 - Successfully linked to http://www.cpan.org.
ok 28 -  . . . and found expected content at http://www.cpan.org
ok 29 - Successfully linked to http://www.campaignfoundations.com.
ok 30 -  . . . and found expected content at http://www.campaignfounda
+tions.com
[download]

and its diagnostic output to: /tmp/test_sites_output_diag, which reads in part:

# Skipping tests of links at: www.perlmonks.com.
# Skipping tests of html validity at: www.perlmonks.com.
# The site is www.example.com
# The hash key is site_www.example.com
#   Failed test 'Successfully linked to http://www.example.com.'
#   at lib/Test/MonitorSites.pm line 216.
#          got: ''
#     expected: '1'
#   Failed test ' . . . and found expected content at http://www.examp
+le.com'
#   at lib/Test/MonitorSites.pm line 217.
#                   '500 Can't connect to www.example.com:80 (connect:
+ timeout)
# '
#     doesn't match '(?-xism:This content is not on this non-existant 
+site)'
# Skipping tests of links at: www.example.com.
# Skipping tests of html validity at: www.example.com.
# The site is validator.w3.org
# The hash key is site_validator.w3.org
# The site is www.cpan.org
[download]

Now to throw an additional wrinkle into the mix,

time prove t/12_exercise_test_sites_method.t
t/12_exercise_test_sites_method....ok 17/45Confused test output: test 
+18 answered aftertest 31
Confused test output: test 19 answered after test 32
Confused test output: test 20 answered after test 33
# Looks like you failed 2 tests of 45.
Confused test output: test 21 answered after test 34
Confused test output: test 22 answered after test 35
Confused test output: test 23 answered after test 36
Confused test output: test 24 answered after test 37
Confused test output: test 25 answered after test 38
Confused test output: test 26 answered after test 39
Confused test output: test 27 answered after test 40
Confused test output: test 28 answered after test 41
Confused test output: test 29 answered after test 42
Confused test output: test 30 answered after test 43
Confused test output: test 31 answered after test 44
t/12_exercise_test_sites_method....dubious
        Test returned status 2 (wstat 512, 0x200)
DIED. FAILED tests 17-30
        Failed 14/45 tests, 68.89% okay
Failed Test                       Stat Wstat Total Fail  List of Faile
+d
----------------------------------------------------------------------
+---------
t/12_exercise_test_sites_method.t    2   512    45   14  17-30
Failed 1/1 test scripts. 14/45 subtests failed.
Files=1, Tests=45, 487 wallclock secs ( 1.44 cusr +  0.10 csys =  1.54
+ CPU)
Failed 1/1 test programs. 14/45 subtests failed.

real    8m8.100s
user    0m1.603s
sys     0m0.134s
[download]

Any ideas on how to proceed here would be greatly appreciated.

-- Hugh