Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery
 
PerlMonks  

Re^4: Neither system testing nor user acceptance testing is the repeat of unit testing (OT)

by leriksen (Curate)
on Oct 24, 2005 at 01:53 UTC ( [id://502356]=note: print w/replies, xml ) Need Help??


in reply to Re^3: Neither system testing nor user acceptance testing is the repeat of unit testing (OT)
in thread Neither system testing nor user acceptance testing is the repeat of unit testing (OT)

OK - in reponse to your challenge, I've got one word to say to you - Exception Flows ;-)

Say you have a function/method, as part of your interface, that writes a file out somewhere. This isnt the main role of the function/method, its just part of the work it does for you. The normal flow is to open the file, write whatever, and close the file.

But there is also code to deal with being unable to open the file, not all the data being written or the file not being able to be closed.

How these failures get communicated back to the client can vary too - from not saying a word through to throwing an exception or even calling exit().

My opinion is that the developer is required to setup test cases for each of these failure modes, and this requires a white box approach. Knowing that the code writes a file may not be inherently knowable from the interface (whether it should be is a different point), but only by looking at the code do you _know_ that it writes a file, and that it does different things for different failures. Even if the interface doco states that it writes a file, detailing the different failure paths is almost certainly not part of the interface design or doco (but may well be part of the interface _implementation_ doco).

The function/method writer can test all the normal stuff just by reference to the interface, but setting up file-open failures and checking that the function does the right thing needs more detailed knowledge related directly to the code. This is where tools like Devel::Cover can help - exception flows pretty much stick out straight away as bits of code, related directly to your chosen implementation of the interface, that you have got to write test cases for.

...reality must take precedence over public relations, for nature cannot be fooled. - R P Feynmann

Replies are listed 'Best First'.
Re^5: Neither system testing nor user acceptance testing is the repeat of unit testing (OT)
by dragonchild (Archbishop) on Oct 24, 2005 at 13:19 UTC
    Ouch. Any possible exception that might be thrown by you needs to be listed in your interface, because it's a possible "return value". In addition, you also need to list every single interface that you use, because your caller might need to look up their interfaces to see which exceptions they might throw.

    To recap, exceptions most definitely are part of the interface spec.


    My criteria for good software:
    1. Does it work?
    2. Can someone else come in, make a change, and be reasonably certain no bugs were introduced?
      I'm not limiting myself to exceptions and throws - I'm talking about exception flows - parts of the cod that gets executed when something goes wrong. This may result in an exception getting thrown, but only if the designer (probably not me) says I can throw - should the design be at that level of polish. Maybe the designer says - thou will not throw an exception in your implementation - I might disagree, but thats what I _have_ to do. I dont think I'd have a job long if I said "screw you , I'm throwing one anyway."

      What I am referring to are testing the decision I made for the implementation of the design. _If_ I chose to use an on-disk file as part of my solution, and _if_ I wrote code to handle different I/O failures, I would want to make sure I wrote unit test cases to check I correctly implemented handling the different failures. Maybe I just handle them and dont tell the caller anything, maybe I log it and continue on. Whatever I chose to do, I should test I implemented correctly. How wise my on-disk solution is, that's a different question.

      In short, those things I can test from the interface/design point-of-view, I agree with you completely and I should test black-box, those things that are artifacts of my implementation, but not part of the interface/design, I should test black-box if I can, and white box where I need to. Its all about making sure you tested "everything" - well it is for me, unit testing my code to as close to 100% coverage as possible is what I aim for. In the OP's case, calling a bad SQL wasnt something their testing had caught. Somewhere, if they ran a coverage report, they might see where that SQL was being injected, as a piece of code that hadn't been tested - a very white box approach.

      ...reality must take precedence over public relations, for nature cannot be fooled. - R P Feynmann

        Much as I typically disagree with dragonchild, I'll agree with him on this point.

        Your caller should know what the impact of calling your function is. If it has side effects or dependencies (such as requiring sockets to be available, stomping on global variables, requiring filesystems to be writeable, or so forth), those restrictions should be published as as part of the function interface, so that they can be tested, by someone other than the author of the code.

        If a tester can get an undocumented side effect from calling a given function, that's a bug in the code, or the code documentation. Either is bad.

        How much this matters depends on what you're dealing with, and what failure modes you can handle. If, for example, you're sure your production server can't possibly run out of filehandles, because it will crash due to other limitations long before that becomes an issue, perhaps there's no real viable way to test, let alone recover from the "out of filehandles" condition. If you document your position, it's at least a calculated risk, not just an accidental assumption, and if it proves to be false, you at least have a explanation for why the choice to ignore it was made. And if/when you upgrade your server, you'll catch the limitation of the software when you read through the limitations list for all your server code prior to the upgrade.
        --
        AC

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://502356]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others chilling in the Monastery: (7)
As of 2024-03-28 12:59 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found