http://qs1969.pair.com?node_id=477243

punkish has asked for the wisdom of the Perl Monks concerning the following question:

For various reasons (frustration and limitations with existing modules, self-edification) I decided to write a wiki formatting module. It is a deceptively easy thing to do -- html is a can o' creepy crawlies.

Concurrently, I have an increasing level of interest in testing. So, I read everything I could lay my hands on -- the PODs for Test::Simple and Test::More, the slideshow on Petdance, and the requisite chapters in various books.

I have come to the conclusion that I somehow need to utilize WWW::Mechanize, HTML::Lint and perhaps one of the afore-mentioned testing modules to test the robustness of my hereto lame-ish but satisfying wiki formatting module.

The problem is, I have no idea how to do that. Basically, I don't want to test for things I know. I want to test for all the possible arbitrary text that users could throw, and see if my module can format it to valid html in the manner that the users expected it to do.

Would any of the knowledgeable monks (all monks are knowledgeable, some more so) kindly point me to a tutorial that does something like what I want to do? Or, could a generous monk consider putting up a tute like so?

Mérci beaucoup.

--

when small people start casting long shadows, it is time to go to bed

Replies are listed 'Best First'.
Re: Wanted, more simple tutorials on testing
by davidrw (Prior) on Jul 22, 2005 at 15:28 UTC
        Basically, I don't want to test for things I know.

    Yes, you do. The edge/weird/invalid cases all need tests, but so do the standard simple cases. What this coverage (of the simple/known stuff) will do is ensure that the basic blocks don't change behavior. It also provides finer-grained resolution for debugging when tests fail -- for example if a weird edge case starts failing, you're going to assume it's because it's a weird edge case and the edge-case code has a problem, but what if it's because you tweaked a core function and it's just rearing it's head here? Good tests on that core function would point to that immediately.

    As for your actual test-creation, my first thought is that I'm not sure you need (or want) to use WWW::Mechanize (note also the existence of Test::WWW::Mechanize) .. for one, it requires a web server running and is only needed for testing the gui screen functionality, but here it seems like you really want to get tests down for the back-end functionality.

    For your actual individual tests, it seems like you need 1) input text 2) if the result against HTML::Lint should be valid or invalid and optionally 3) a string to compare the result against
    You can set that up in a data structure, and then iterate over it performming the tests...

      I'll second this. Your code is still just code -- it only does what it's told. Write your tests to define what you want it to do -- and that includes handling unexpected stuff. The hard part is actually defining what you want it to do, not writing tests.

      At the granular level, much of your code converts a string of input to a string of output. Set up a test loop that does that and define your inputs and outputs in a data structure. You don't need to worry about the web parts yet -- test at a simpler level. E.g.

      use strict; use Test::More; my @cases = ( { input => "= Headline", output => "<h1>Headline</h1>", label => "Single =" }, { input => "== Headline", output => "<h2>Headline</h2>", label => "Double =" }, ); plan tests => 1 + @cases; require_ok ("My::Module"); # assume exports 'convert' for my $c (@cases) { is( convert( $c->{input} ), $c->{output}, $c->{label} ); }

      Once you've got a framework like that in place, just keep adding tests for all the basic, most granular elements. Then I'd suggest considering more complicated cases in similar groups -- e.g. nesting of items inside each other, ordering of nested tags, missing whitespace, etc. (A good time for a separate test file.) Once you have a category, it should be easier to imagine lots of variations. The key is to start small -- focus on small, simple combinations before working up to larger units.

      You might also try thinking about the problem in terms of a state-tree. As your code receives each chunk of input, it enters certain states. Are you testing each of the states that different chunks of input lead it to?

      Then get malicious. Intentionally try to break your code or make it give improper results -- which is when you may want to use Test::Exception to see if how errors are handled, too. You know your code better than any random end-user so you should be much more likely to generate malignant input than the "million monkeys" that might bang on your code later. (You'd be surprised how often ideas for this will occur to you during the simple tests of expected behavior.)

      The point of all of this is to focus on exploring the desired behaviors of your code, both when input is as expected and when it's not. If you do that well at a granular level, then any sort of "real world" input is muchlikely going to fit some pattern you've already tested.

      -xdg

      Code written by xdg and posted on PerlMonks is public domain. It is provided as is with no warranties, express or implied, of any kind. Posted code may not have been tested. Use of posted code is at your own risk.

Re: Wanted, more simple tutorials on testing
by davorg (Chancellor) on Jul 22, 2005 at 15:14 UTC
Re: Wanted, more simple tutorials on testing
by rinceWind (Monsignor) on Jul 22, 2005 at 15:18 UTC

    Sorry I can't offer you a tutorial, but take a look at the tests that come with OpenGuides and the CGI::Wiki family of modules. You should be able to find examples of the kind of scenarios your wiki formatter needs to cope with, especially as we were doing very much the same job.

    This highlights an interesting design point, in that if you separate the text munging from the web stuff, you can write tests that don't rely on having Apache running, let alone configured.

    Basically, I don't want to test for things I know. I want to test for all the possible arbitrary text that users could throw, and see if my module can format it to valid html in the manner that the users expected it to do.

    That's a toughy. You DO want to write tests for things you know - this is a starting point. In terms of all possible text, this is fairly intractible. Consider the documentation. The pod that goes with the module gives examples of how to use it - these examples should be included in the tests. For the more esoteric scenarios, my advice would be to get a project team or an on-line community together, inviting feedback from as many tech-savvy users as possible. Every time you find something that misbehaves, write a test for it.

    Good luck with your module

    --

    Oh Lord, won’t you burn me a Knoppix CD ?
    My friends all rate Windows, I must disagree.
    Your powers of persuasion will set them all free,
    So oh Lord, won’t you burn me a Knoppix CD ?
    (Missquoting Janis Joplin)

Re: Wanted, more simple tutorials on testing
by ww (Archbishop) on Jul 22, 2005 at 15:48 UTC
    and, albeit non-perlishly, not as an alternative to the above suggestions, but as a step beyond, I would strongly that you test at least some of your html against the w3c validator http://validator.w3.org/. If you use css (recommended), w3c also offers a reasonably trustworthy validator for that.

    html is and css can be made sufficiently english-like that it's fairly easy to scan for correctness (certainly it's easier for me than debugging my sorry excuses for scripts). Your creepie-crawlies hide chiefly in the various (and disparate) non-standard performances by browsers. However, a little time invested (say, a few hours) in learning these will let you write html and css that perform reasonably well across (BIG caveat) reasonably current (say, v5 and up in the windoze world) browsers.

      Better than that, there's HTML-Tidy. It's basically the same as the w3c validator, but much faster, and offline : you don't have to make your file accessible to the whole net :) It even can correct most common mistakes (unclosed tags, deprecated tags, unencoded entities, bad encoding...)

      Get it there : HTML Tidy project
        Better than that, there's HTML-Tidy. It's basically the same as the w3c validator,

        While tidy is certainly a useful app it is nothing like the W3C validator. It doesn't do verification against doctypes, doesn't recognise use of meta tags to set character encoding, etc.

        adrianh is completely correct; tidy (HTML) is an excellent tool for partial cleanup of bad (ie, non-w3c compliant) html and for pretty-fying code to make it easier to read when another author (or one of the less sensible WYSIWYG) has obfuscated the code... but it is NOT a validator.

      It isn't a "non-Perlish" suggestion. The W3C validator is written in Perl, last I looked.

Re: Wanted, more simple tutorials on testing
by DrWhy (Chaplain) on Jul 22, 2005 at 16:26 UTC
    The problem is, I have no idea how to do that. Basically, I don't want to test for things I know. I want to test for all the possible arbitrary text that users could throw, and see if my module can format it to valid html in the manner that the users expected it to do.

    While getting a community of smart users to help you test can be useful, it may be hard to come by, especially when you are dealing with code that does a task very similar to one that several other modules do. As an alternative, you might want to explore the ideas of stochastic/random/specification-based testing. In very simplistic terms, this is (if my limited understanding is correct) a way to do tests with input data that is (pseudo)randomly generated but within a specified set of constraints. You might start by looknig at the Test::LectroTest module. There are a number of resources on this concept out on the net. The URL, http://www.acmqueue.com/modules.php?name=Content&pa=showpage&pid=277, seems to keep coming up near the top in my Googling, so it might be a good place to start.

    --DrWhy

    "If God had meant for us to think for ourselves he would have given us brains. Oh, wait..."

Re: Wanted, more simple tutorials on testing
by adrianh (Chancellor) on Jul 23, 2005 at 09:50 UTC