1. I have found the weirdest stuff break, stuff I would never have imagined.
2. It's a form of documentation, for those of us without documentation sections in our non-existent project plans.

The only warning I'd have is that whenever a test fails, you have to make sure it's testing what you want it to. Recently, I made a necessary non-backwards-compatible interface change. Worked fine. Then, I started using it. Made a few more changes, then BAM! a test started failing. I looked deeper and found it was a test I hadn't updated when I made my interface change. The test hadn't been testing what I thought it had.

It's a form of documentation, for those of us without documentation sections in our non-existent project plans.

Amen to that brother dragonchild! Although more times than not for us, the "documentation" phase is usually right before/after the "testing" phase, and both of them are in the always nebulous "last few weeks" of the project. I have been thinking about pushing the Test Driven Development Philosophy as a way to avoid this all too common scenario, and get both tested code and (pseudo) documentation in one fell swoop.

-stvn

Amen to that brother dragonchild!

Test-ify! Test-ify!
(Shouldn't this thread be titled "Dr. Testlove..."?)

---

The PerlMonk tr/// Advocate

After reading an article on it in TPJ and trying it out, I strongly recommend checking out Devel::Cover. It automates the process of seeing how much of your code is actually tested by your test suite and produces a fantastic HTML drill-down report. It will track the execution of statements, branches, conditions, subroutines, and pod.

Perhaps one day this will even lead to test-driven test development for those of us who are too lazy to do test-driven development itself -- i.e. what's the minimal amount of test suite I need to write to actually exercise the code I've already written?

Excellent suggestion. The question of "what do I test?" should always be answered with "what have you tested"? and Devel::Cover does a great job of that. I've wowed coworkers with its HTML output :)

<malden type="karl">Devel::Cover. Don't leave home without it.</malden> (have I just dated myself?)

Cheers,
Ovid

New address of my CGI Course.

Ovid

I am actually running some of my tests through Devel::Cover right now, and and something just popped up.

What about on coverage for things like this:

    return wantarray ? @results : \@results;
[download]

Devel::Cover reports that I have only tested the @results return value branch. Should I test the \@results branch too, or is this being too granular?
-stvn

Yes, Devel::Cover is one of my favorite new toys too. I just actually showed it to my boss to demonstrate exactly what you are saying, how we can test test-coverage. He was really impressed, it was one more thing to help me in selling this way of testing.

Do you happen to have a link to that TPJ article? I actually read about this module on CPAN (saw it in the CPAN nodelet one day and was intrigued), but i would love to read the article.

-stvn

Unfortunately, TPJ is subscription-only now and doesn't seem to put archives online anymore since they went to a subscription model. (Even for subscribers! I can only find a link to the previous month, not historical months.)

-xdg

P.S. (Added 4/1 and not a joke): For those looking for more info, Paul Johnson, the author of Devel::Cover, has an paper and slideshow from YAPC::Europe::2002. There's also a very interesting look at coverage on various CPAN modules.

Code posted by xdg on PerlMonks is public domain. It has no warranties, express or implied. Posted code may not have been tested. Use at your own risk.

I test every method before its called with can_ok, and I (re)test the same method on each instance I create. Sometimes I actually set up test_Object_Interface functions and use it to test subclasses (which is the source of many of the duplicate tests).

I would not do this. If the methods are called once and succeed, that should be enough. Testing with can() is not very useful, since that isn't really calling the interface to your class.

I test all my constants using can_ok and then test their value.

I don't see how you could do that without embedding the values of the constants in your test, which would be wrong. Unless these constants are a part of your public API, you should not be testing them at all.

Redundancy is not something you want in your code, and tests are code.

But most of the redundancy of the tests is actually encapsulated in functions (much as you would do to deal with regular code redundancy). I fear that to try and remove the redunancy would result in my not being able to use the functions (test would get to specialized) and while it would reduce redundant tests, it would decrease the modularity of my test code. Is this still not ok?

I test every method before its called with can_ok, and I (re)test the same method on each instance I create.

I would not do this. If the methods are called once and succeed, that should be enough.

Yeah this one seemed like I was over the edge. Thanks for confirming that.

Testing with can() is not very useful, since that isn't really calling the interface to your class.

I am actually not using can, but Test::More::can_ok which no doubt it implemented in terms of can (though I have not delved into the source code to be sure, and the difference is probably neglible). I do have to disagree that can is not "really calling the interface to my class" though. Sure it can be calling a subclass potentially, but IMO thats the class interface (just inherited). I view these tests in particular (the Interface ones), as a means of helping me keep my API straight across versions/updates. It makes no assumptions about implementation, but only that the API is the same. It has already proven useful in tracking down a bug in an older installation of our framework, by finding where an API had changed between versions, but had gotten missed while someone was making changes to the code which used the framework. Why do you feel this is not "really the class interface" I am interested to know?

Unless these constants are a part of your public API, you should not be testing them at all.

They actually are part of the public API, which is why I wanted to test them.

I don't see how you could do that without embedding the values of the constants in your test, which would be wrong

Very true (and exactly what I was doing (Bad programmer! Bad!)), but being part of the public API I want to test them, I suppose that just testing that they are there (can_ok) is good enough. Do you agree?

-stvn

But most of the redundancy of the tests is actually encapsulated in functions

That sounds fine then.

I do have to disagree that can is not "really calling the interface to my class" though.

It isn't calling the interface. It is calling the can() method which is typically inherited from UNIVERSAL::can(). This will break if you do any sneaky AUTOLOAD stuff. (Sure, you could fix that, but what a waste of time.) It's better to actually call the method you are checking instead.

I suppose that just testing that they are there (can_ok) is good enough. Do you agree?

Yes, for constants. I think it's better to use globals than the subs that the constant pragma creates, but that's a whole other discussion.

I would just like to say that redundant tests have some negative qualities:

• The test suite will run slower, and with 2,500 tests, that is going to become an issue.
• Time spent developing unnecessary tests detracts from development of new code.

I have some common sense guidelines for test writing:

• Test the boundary cases far more than the regular cases.
• Focus tests on buggy or expected-to-be-buggy code.
• If you can write the test cases first, then good for you, but be aware that the test suite is broken until the code is completed.
• Interesting things happen when Person A writes tests for Person B's code.
• If you have performance requirements, write tests that only pass if those requirements are met.
• Coverage is important.

pbeckingham

Excellent guidelines. Especially the "Person A writing tests for Person B's code" one. ++

The test suite will run slower, and with 2,500 tests, that is going to become an issue.

To be honest, I am not really concerned about this, since its a rather large framework (almost 150 classes, almost 13,500 loc) and (for us anyway) its the foundation of our applications. It really has to be reliable for us, so even if the tests took an hour to run that would be okay for us. I realize this may not be okay when we get around to distributing this, but I will deal with that when the time comes.

Time spent developing unnecessary tests detracts from development of new code.

Thats just the thing, the tests run are redundant, but alot of them are in functions and therefore really didnt take anymore time to develop. Its actually saving me time, since I can re-use the test even though it is somewhat testing something I know is already tested (sort of).

-stvn

To be honest, I am not really concerned about this, since its a rather large framework (almost 150 classes, almost 13,500 loc) and (for us anyway) its the foundation of our applications. It really has to be reliable for us, so even if the tests took an hour to run that would be okay for us.

Nice in theory. Sucks in practice. More than one developer here (on a "must never fail on pain of death" project) has skipped running the entire 1.5 hour test suite because "their one little change" won't break things and their one little test didn't break. When you're in a hurry, one and one half hours can seem like a long time :)

Redundant tests are bad. They slow things down, they are more tests to maintain and they provide no benefit. You don't need to prove that 2+2 == 4 more than once.

Cheers,
Ovid

New address of my CGI Course.

The test suite will run slower, and with 2,500 tests, that is going to become an issue.

I have serious issues with this line of thought. In my opinion, one should run every test that deals with any piece of code that has a 0.00001% chance of dealing with the code you changed since the last time you ran a full test suite.

Now, my codebase is broken up into 23+ Perl distributions (each its own CVS project) and 4 other CVS projects containing Oracle DDL, templates, and the like. The projects have a definite hierarchy (which is delineated in every distributions Makefile.PL). If I change some code in XXX::Foo and YYY::Foo depends on XXX::Foo, I need to run the tests in both XXX::Foo and YYY::Foo. Period. There's just no way around that.

Now, I don't have to run the tests for XXX::Foo if YYY::Foo changes, and the test suites need to be flexible enough to handle that. But, I think it's foolish to complain about test suite size or the time to run it. You have tests for a reason. Don't sabotage yourself just because you're impatient. It may be a virtue, but so is patience.

------
We are the carpenters and bricklayers of the Information Age.

Then there are Damian modules.... *sigh* ... that's not about being less-lazy -- that's about being on some really good drugs -- you know, there is no spoon. - flyingmoose

Please bear in mind that I was referring to redundant tests slowing the test suite. Redundant is being used here to mean unnecessary. If you are advocating more tests, each of which increases coverage to some degree, then I am in agreement with you.

Redundancy in the test suite - by definition - is pointless, and serves to lengthen the wait every time it is run, and that will multiply up and become, as I said, an issue.

Also, I'd be glad to visit your local Perl Mongers group and talk, too. (No cost to you, either, so long as my costs to get there don't exceed $100) I'm going to Minneapolis in a few weeks, and I've spoken at St. Louis and Grand Rapids.

Short version of my comments on this thread:

• Redundancy is OK. Don't build for redundancy, but if you happen to have redundant tests between a couple of .t files, that's OK. Don't try to remove the "inefficiencies".
• Don't worry how long your tests take. Set up a bot that runs hourly and notifies you (or the entire department, in my case) if any tests fail.

# Runs the smokebot program on the trunk and mails the results
0 * * * * smokebot HEAD smokereports@us.com

# Smoke against the branches
30 * * * * smokebot cp2004-branch smokereports@us.com
[download]

#!/bin/sh

if [ $# -lt 2 ]
then
    echo Must pass at least a branch, and one email address, 
    echo plus any parms to echo pass to smoke.
    exit 1
fi

REV=$1
shift

MAIL=$1
shift

cd $TMP
DIR=tw
FULLPATH=$TMP/$DIR

# This assumes you have already logged in once as anoncvs
# so that the password is in your ~/.cvspass file.
cvs -d/home/cvs -Q co -d $DIR -r $REV tw > /dev/null

TWROOT=$FULLPATH
export TWROOT

/home/smoke/tw/Dev/devapache stop  >  /dev/null 2>&1
/home/smoke/tw/Dev/devapache start > /home/smoke/smoke.out 2>&1 

cd $TWROOT
smoke $@ >> /home/smoke/smoke.out 2>&1 
grep -i "^Failed" /home/smoke/smoke.out > /home/smoke/smoke.out.fail

if [ -s /home/smoke/smoke.out.fail ]
then
    STATUS="FAILED"
    mail -s"Smoke $REV $@ $STATUS `date`" $MAIL < /home/smoke/smoke.ou
+t
else
    STATUS="passed"
fi

/home/smoke/tw/Dev/devapache stop >> /home/smoke/smoke.out 2>&1
[download]

Just a short node to tell you that I am anal enough to test everything as much as can as long as time will allow. Don't pull a MicroSoft and throw it out in Beta form for all to find your bugs. I think that this is unlawful and very annoying (from a users stand point). I also probably test too much. Considering that the code I write usually has something to do with our database, which is the lifeblood of our system, I want to make sure that what I am writing does exactly what I want it to and exactly what I expect it to. In our production environment, if I cause our DB to go haywire, I'm in some serious dudu-caca. (I know, preaching to the choir).

Bottom line from my point of view is that testing is the only way to see what your code is going to do for sure. Just make sure that you don't run over your time budget for your project.

Just thinking that maybe the writers basic rule of thumb isn't such a bad idea. That (as far as I recall) is to put in as much time necessary to complete the book (code), then put in another 10% on top of that improving it.

 
Never underestimate the power of one little test.

There is no such thing as a dumb test.

 Your tests can often find problems where you're not expecting them.

 Test that everything you say happens actually does happen.

 If it's worth documenting, it's worth testing. 

I asked him a few questions on his testing methodology and Andy answered back a few gems that you may find interesting ... (reproduced below).

1. Think about what the code should do.
   
2. Think about the API.
   
3. Write the documentation for the code, explaining what the parms do.
   
4. Write the test code that uses the API.
5. Keep doing #3 and #4 until all the cases are covered. "Oooh, I hadn't thought of the case where a length is negative." So you type in the docs "If the length passed is negative, then a warning is thrown," and then write the test that tests that. Or vice versa.
   
6. Write the code. You may go back to #3 and #4 as necessary because you think of more things as you write.
   
7. Check it into CVS. Move on, knowing that you're covered.

What I get back from this is testing is soft of self discovery processing where testing allows you to enforce how your code should behave. At the same time uncover defects. Follow the links to see what modules are being used.

Some time I need to combine all this stuff into http://qa.perl.org/ so it's all in one place...

xoxo,
Andy

It's not to much.

Look at gcc and glibc, those take as much time to test then to build them.

Or perl itself for that matter. On my machine it takes only a few minutes to build the Perl interpreter, but at least 20 minutes to run the tests.

-- Mike

--
XML::Simpler does not require XML::Parser or a SAX parser. It does require File::Slurp.
-- grantm, perldoc XML::Simpler

It once took me two days to run configure for Perl, 10 minutes to compile, and another day for the tests. :-)

------
We are the carpenters and bricklayers of the Information Age.

Then there are Damian modules.... *sigh* ... that's not about being less-lazy -- that's about being on some really good drugs -- you know, there is no spoon. - flyingmoose

You should probably take a look at the book _The Capability Maturity Model for Software Development_, published by the Software Engineering Institute. Tests are all well and good, but if they don't improve your development process, you're not getting as much benefit out of them as you should. You should also start tracking which specific tests identify bugs each time you run your test suite.

Bugs don't occur randomly, they occur for specific reasons. Your test suite will help you locate the reasons behind the bugs, and then you can deal with each cause in a way that makes sure you never need to test for it again.

Case in point -- I used to forget to write the return() statement at the end of the occasional function:

    package Thingy;
    
    sub new {
        my $O = bless {}, shift;
        
        [lots of value setting here]
        
        [and oops, no return() statement!]
    }
[download]

When I started tracking my mistakes, I discovered that problem, and made a trivial change to the way I write code. Now, every function I write starts off looking like so:

    sub func {
        return ($result);
    }
[download]

Then I go back and fill in all the code necessary to generate $result. That simple habit has saved me no end of headaches and bug fixes. I also put my constants first in conditionals:

    if (3 == $x) {
        [whatever]
    }
[download]

because typing '=' instead of '==' will create a syntax error, not a valid (and invisible to human inspection) assignment statement. (Admit it, you had to read that last sentence twice to see how the two strings were different, didn't you? ;-)

Writing tests is like optimizing your code. Yes, it's fun, but you can invest massive amounts of effort in it, and a test that doesn't locate bugs is a waste of effort. 80% of the payback will come from 20% of your test suite, so figure out which 20% that is, and focus your efforts there.

Create a list of tests that have actually detected bugs at some point in their existence, and use that set as your day-to-day test suite. Then run the whole megillah once every week or two just to see if anything new has crept in. When you write a new test, put in the big once-a-week set, but don't move it to the daily-use set until it actually locates a bug.

Once you do find a bug, try to find a way to improve your coding practices so you don't keep repeating the same mistake over and over again. Keep running the relevant tests for a week or two to see if the improvement has worked, and if it does, demote that test back to the once-a-fortnight set.

That will keep your basic test suite slim while still giving you the option of being comprehensive, but it will also help you zero in on which parts of your code really need to be tested.

Like everything else in the world, it depends. If you need a specific environment which is non-standard, and needs to be set up properly to do tests, you may just need 1 test to ensure the data loads, and then provide examples of how to use the module. (It's amazing to me anyway how adjusting the documentation breaks things.) Otherwise, it's a good idea to write tests when a lot of people will be using the code. The only other gotcha I've come across is in emergency situations and you need an application 'now', then you may skip the testing to get the capability working, and then go back and ensure that you have valid tests for the code you wrote...

I think the only danger of adding too many tests is that the test suite will take too long to run, and so will be run less often or slow down development. A big part of the point of automated testing is that the tests are run frequently so bugs are found quickly, before too many changes have been made and while the code that introduced the bug is still fresh in your mind. If the test suite takes more than a few minutes to run, it defeats the point of this.

I agree, I have actually removed much of the redundant tests I spoke of, and am considering moving to cpan:://Test::Class instead anyway. Although, I have found while developing another suite for another application, that using the prove utility that comes with Test::Harness comes in handy, I can easily run a specific set of tests or a single test and therefore reduce testing time during development. Then of course when I feel I need to I can run the whole suite. So far it has worked well for me.

-stvn

I'm new to this testing piece, but enthusiastic about what I've been learning.

Tonight, I'm committing my latest project's access control test script which became the subject of two or three questions to the SoPW page and a couple of days of distraction as I did the research to help me understand what it is I was doing. I'm using WWW::Mechanize and testing lots of interactions with the website. As I muddled through the logic of some of the access control rules, I found myself commenting out four of my five test cases from the hash of test cases and two of the three subroutines I processed that test data through. My 168 test suite usually ran only a dozen or so tests each time (except of course that it error'd out after the fifth one a good bit of the day).

But then I had my Eureka moment and made it to the sixth test, and with a little more clean up through the limited number allowed by my commented out logic in the main loop. I deleted my comments and let the whole suite run, not just the latest, but all eight .t files added so far. I didn't mind the wait as 05access_control.t used LWP to exercise the user interface. It put it through its paces. I checked my email, stretched, checked out what was new at perlmonks and finally reviewed the scroll buffer for the ok messages I had missed.

My eight test scripts took six and a half minutes to completely run, to where they error out now. But it was certainly sweet to think, with as much work as still lays ahead, that at least that much of the application is already working.

Me? I look forward to the day when I've added so much functionality to this application that I would have an hour of tests to run to fully exercise it. By then I'm sure that I too will have adopted the practice of cron driven test runs.

-- Hugh