hardburn has asked for the wisdom of the Perl Monks concerning the following question:

In one of merlyn's columns on Checking website health, he shows how to use Test::More's SKIP blocks to skip tests that rely on a previous test passing. For instance, before testing that the title of the returned HTML is accurate, we should make sure that the request was successful and HTML is being returned, and if not, we might as well not check the title.

However, I'm finding that a significant number of new blocks is needed to fully describe the correct behavior:

# $mech is a previously-intitlized WWW::Mechanize # instance currently holding the page I want to test SKIP: { ok( $mech->response->is_success, "Request is successful" ) or skip "Unsuccessful request", 4; SKIP: { ok( $mech->is_html, "Returns HTML" ) or skip "Didn't return HTML", 3; ok( $mech->title eq 'some title', "Returns expected title" ); my @forms = $mech->forms; SKIP: { ok( @forms >= 1, "Has at least one form" ) or skip "Need at least one form to wor +k on", 1; my ($passwd_form) = @forms; ok( $passwd_form->action eq 'example.cgi', "action goes to correct CGI" ); } } }

That's a lot of nesting, and I haven't gotten to the really important parts yet.

In noticed that in Part II of merlyn's article, he breaks the third SKIP block I show above out of the nesting and into the top-level block. However, I believe that the code I show above is the more correct approach, being that we can't assume there will be any forms if the request was unsuccessful or if it didn't return HTML.

Possible solutions:

  1. Break out some of the nested SKIPs, as merlyn showed. I think this goes around the problem.
  2. Don't use SKIP at all and let dependant tests fail. This is the solution I'm leaning twards for the immediate project at hand, though it also sidesteps the problem.
  3. Consider Test::More's SKIP methodology to be broken and develop a more scalable solution. This is the way I'd like to go, but I have a hard time justifying the time it would take for the project I am currently working on.

Is there a practical way to use the current state of Test::More without side-stepping the problem? Has suggestion #3 already been done and I just haven't dug through CPAN enough to find it? If no to both the above, does anyone have suggestions on a better way to implement heavily-nested skip tests?

Update: Removed redundant is_html test.

----
: () { :|:& };:

Note: All code is untested, unless otherwise stated

Replies are listed 'Best First'.
Re: Keeping SKIP Test Blocks Under Control
by Ovid (Cardinal) on Apr 12, 2004 at 16:54 UTC

    Let the dependant tests fail and fix them from the top down. Use skip when you can't test a particular bit of functionality.

    SKIP: { skip 'Not on Linux', 1 unless $os_is_linux; ok( linux_specific_test(), $test_name ); };

    Cheers,
    Ovid

    New address of my CGI Course.

Re: Keeping SKIP Test Blocks Under Control
by halley (Prior) on Apr 12, 2004 at 17:03 UTC
    I noticed that your skip blocks all seem to be organized for fail-fast, that is, an initial test would bail all of the remaining tests.

    Since ok() appears to return whether it was okay or not (not clearly documented; see source for Test::Builder::ok()), just last if not ok(...) ahead of any dependent tests. Now everything's at the same scope/level, instead of nested deeply.

    There's also no harm in having more than one test script for a given module. I break up complicated testing into independent regions of functionality. Each rare situation where the tests can't be organized to use last if not ok(...) is a good candidate for starting a new script with its own sequence.

    Lastly, I guess I don't see much value in the percentages and count of tests planned. It either failed to execute the test script, in which case perl will croak for you, or it succeeded in running the test script. It either failed a test, or all tests were ok(). Finding out that your module passes 93% seems of dubious or anecdotal value. So don't worry about messages to the effect of "script seems to have run 45 fewer than planned."

    --
    [ e d @ h a l l e y . c c ]

      just last if not ok(...) ahead of any dependent tests

      That breaks the test plan, which you are correct in saying that it doesn't have a lot of value, at least from a development point of view. However, it also has some value when your boss comes to your cube and asks what you're up to :)

      Even so, some of the other posts show that skip doesn't have to be nested at all, which works out the same as last if not ok but maintains the test's plan.

      There's also no harm in having more than one test script for a given module.

      In this case, it's a CGI::Application program and each page returned is strongly related to what was filled into the previous page's form.

      ----
      : () { :|:& };:

      Note: All code is untested, unless otherwise stated

Re: Keeping SKIP Test Blocks Under Control
by adrianh (Chancellor) on Apr 12, 2004 at 17:15 UTC

    Good news! You don't have to nest the skip blocks ;-) Something like:

    SKIP: { ok( $mech->response->is_success, "Request is successful" ) or skip "Unsuccessful request", 4; ok( $mech->is_html, "Returns HTML" ) or skip "Didn't return HTML", 3; ok( $mech->title eq 'some title', "Returns expected title" ); my @forms = $mech->forms; ok( @forms >= 1, "Has at least one form" ) or skip "Need at least one form to work on", 1; my ($passwd_form) = @forms; ok( $passwd_form->action eq 'example.cgi', "action goes to correct CGI" ); }

    should work just fine.

    Also, you could always just not have a test plan and use a plain return to skip unnecessary tests.

    Finally, another alternative would be to use something like Test::Block which would save you having to keep track of the number of remaining tests in the skip block.

    In fact - I should really go release Test::Block to CPAN since its actually fairly useful. I'll dig it out and you should be able to use it later tonight :-)

      ok( $mech->is_html, "Returns HTML" ) or skip "Didn't return HTML", 3;

      That makes me want to write Test::Skip that wraps ok(), is(), like() and so on to avoid having to duplicate the test name. It'd be:

      ok_or_skip( $mech->is_html, 'Returns HTML', 3 );
        That makes me want to write Test::Skip that wraps ok(), is(), like() and so on to avoid having to duplicate the test name.

        Hmmm. Interesting thought. I think you could just get away with something like this:

        sub skip_unless (&$$) { my ($test, $name, $tests_to_skip) = @_; local $Test::Builder::Level = $Test::Builder::Level+1; my $passed; my $ok = \&Test::Builder::ok; { no warnings; local *Test::Builder::ok = sub { $_[2] = $name unless defined $_[2]; $ok->(@_); }; use warnings; $passed = $test->(); }; skip "failed: $name", $tests_to_skip unless $passed; return $passed; };

        allowing you to do things like:

        skip_unless { ok $mech->is_html } 'Returns HTML', 3; .... skip_unless { is $foo, $bar } 'foo == bar', 3; ...

        So you wouldn't have to have N different *_or_skip routines. Not sure whether this would deserve a module all of its own though...

Re: Keeping SKIP Test Blocks Under Control
by fizbin (Chaplain) on Apr 12, 2004 at 17:21 UTC
    Why are you nesting your SKIP blocks? (I know, "because merlyn did it" -- that alone is insufficient reason) It appears that if a skip happens, you just skip all the way to the end. The only thing SKIP blocks are used for is Test::More's skip function, which just does some accounting and then last SKIP.

    The code you posted is equivalent to this code:

    # $mech is a previously-intitlized WWW::Mechanize # instance currently holding the page I want to test SKIP: { ok( $mech->response->is_success, "Request is successful" ) or skip "Unsuccessful request", 4; # ignoring repeated is_html test in original post ok( $mech->is_html, "Returns HTML" ) or skip "Didn't return HTML", 3; ok( $mech->title eq 'some title', "Returns expected title" ); my @forms = $mech->forms; ok( @forms >= 1, "Has at least one form" ) or skip "Need at least one form to work on", 1; my ($passwd_form) = @forms; ok( $passwd_form->action eq 'example.cgi', "action goes to correct CGI" ); }
        It's ok. You're allowed to be human.

      Why are you nesting your SKIP blocks?

      Because I wasn't sure if repeated skip statements would work and I couldn't find any examples of implementing it without nested blocks. Though now that I think about the implementation details of Test::More::skip, this probably should have been obvious.

      # ignoring repeated is_html test in original post

      Oops. I was cleaning up and sanitizing the code that I'm actually developing for a project, and that one sliped through.

      ----
      : () { :|:& };:

      Note: All code is untested, unless otherwise stated