http://qs1969.pair.com?node_id=1154376

mrguy123 has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks
I am part of a bioinformatics team working in Perl and we are now trying to improve our code standards.
One of the things we are looking to do is run automatic tests (e.g. Test::More) for our code.

A discussion arose on what should we test automatically.
In my previous (limited) experience regular unit testing (or edge case scenarios) would be done manually, and automatic testing would be more global.
That means you wouldn't use Test::More to check every possible outcome of a subroutine but only a a few specific tests to make sure the program is running correctly. Is this the correct approach or am I completely wrong?
Also, what would be the best method for automatic testing of scripts we use?

I understand this is a complicated issue with many different approaches but any feedback would be great!

Thanks for your help (and wisdom)
Guy (MrGuy) Naamati UPDATE: Thanks for the replies - we will take them into account in developing testing standards

Replies are listed 'Best First'.
Re: Testing in Perl
by QuillMeantTen (Friar) on Feb 04, 2016 at 12:14 UTC

    Greetings,
    If you are not using any version control system I'd advise using git. Most of my own code is under git for multiple reasons (easy forking, easy patching and such).

    But we are talking about testing, what does git have to do with it (you ask)
    Easy, my friend: instead of a cron job you can use git hooks. To put it simply hooks allow you to run scripts when some action is done in a repository (such as commiting changes or merging). you can even use those on the server side!

    On the topic of testing, in all my uni projects (and most of my personal ones) I have striven to attain a the best possible test coverage and I found one method to work great:
    test driven development
    if you use that method and hooks, sure you will feel like you are cranking out features at a slower rate but you WILL reap the benefits in the long run (less debugging). And if you have regression issues you can always write a test for that and then use git-bisect to find the first failing commit...
    On the topic of using modules, after I read the second (iirc) famous book on perl wizardry I kinda fell in love with module-starter...

Re: Testing in Perl
by Discipulus (Canon) on Feb 04, 2016 at 11:51 UTC
Re: Testing in Perl
by Myrddin Wyllt (Hermit) on Feb 04, 2016 at 14:24 UTC

    I would say that your test script should test every possible outcome where that is feasible. It should test everything that can reasonably be tested, and particular emphasis should be given to checking the things that are obviously bound to happen do in fact happen.

    If you are new to writing tests, this might sound like a nit-picking waste of time. Having spent some years qa testing code written by far more talented developers than myself, I think that it's definitely worth the extra effort. Just about any code a programmer thinks is ready for testing will (appear to) work fine in most common use cases, otherwise it wouldn't have been submitted. Obviously you should still test for this, but bugs are more likely to be unearthed in corner cases, or in situations where the program isn't doing what it looks like it's doing. (If that doesn't make sense to you, start writing some rigorous test scripts and it soon will).

    If you have the resources, aim to have additional test scripts written by someone other than the author and based purely on the documented interface. This not only results in a test unbiased by the implementation, but will quickly improve the quality of your documentation.

    You can also look at the test scripts for some large CPAN based distributions with multiple authors (one of the web frameworks, for example), to get an idea of the scope and depth of testing that is possible.

Re: Testing in Perl
by stevieb (Canon) on Feb 04, 2016 at 15:13 UTC

    I'm a firm believer of testing every single change against the entire codebase. Here's what I'd recommend:

    • immediately get your code into a version control system
    • write some basic tests for all subs. Don't worry about edge cases initially, the thinking here is that in the interim, you'll just want to spot if a change breaks basic functionality
    • after all subs are done, use Devel::Cover, and go through the parts that don't have tests (line-by-line, condition-by-condition etc is what I do), and try to hit every scenario
    • now that you've got a baseline, go back and add more tests for your subs... make them very obscure (edge cases). Throw bad arguments, too many arguments, arguments out of order etc. Effectively, you literally want to try to break each sub
    • now, you've got a very strong foundation, you can rest a lot easier knowing that you've got your bases covered for code changes. At this point, I'd go back to revisit the Devel::Cover results, and focus on changing the code where it is impossible to trigger certain items to make them reachable
    • you now have your base test platform. Now using a VCS hook as others have mentioned, throw it at a test platform. Recently, I wrote RFC: Continuous Integration (CI) for your CPAN modules (for free!), which explains doing automated testing (Continuous Integration, aka CI) using tools online. Note however, that for private software, nominal fees can be paid and you can use the same procedure, and never have to worry about setting up a repository infrastructure or a testing infrastructure. It just works. *All* of your tests are run automatically on each push to the VCS, so you know immediately if something's broken

    Now, get into the habit of writing new tests for new functionality (or functionality changes) before you write the new code. I do this so I can work out how the new functionality should behave at a basic level. Of course, all these tests will fail until you get the code written. You'll also end up changing/adding tests as you go, which is a good thing. At minimum, as soon as you write something new (or change something), immediately afterwards write tests (and documentation). Don't put it off, or it just won't get done.

    I wrote a script a while back which I'm currently changing into a CPAN module that automagically sets up a full build/test platform using Perlbrew (and Berrybrew on Windows), using VCS hooks, where you can specify different perl versions, have the software select the versions randomly, or test on all perls. Each run builds the entire Perlbrew infrastructure from scratch, so each build is tested in a pristine environment. Later today, I'll at least share the script (I'll post it in Cool Uses for Perl and link to it in this thread). I don't expect the module to be done before next week (at minimum).

Re: Testing in Perl
by jeffa (Bishop) on Feb 04, 2016 at 18:50 UTC

    "That means you wouldn't use Test::More to check every possible outcome of a subroutine but only a a few specific tests to make sure the program is running correctly. Is this the correct approach or am I completely wrong?"

    It is the start to some kind of a correct approach. The point of math proofs is that you don't have to show every value of X or Y, etc. You can treat unit testing the same way with a few caveats, the main one being bug fixes. When someone reports a bug, write (add) a unit test to recreate the bug. The test should fail. Fix the bug. Include the bug report number in the name of the test.

    The flip side is writing too many tests, especially if they take too long to run. Developers will stop running them and they will go to the weeds, becoming a hindrance. You have to keep your unit tests current and they need to finish in a timely manner.

    "Also, what would be the best method for automatic testing of scripts we use? "

    I have my own personal Jenkins server that i use to test and build my distributions. I also use github and travis-ci which are free services. The gist is you use github as a repository for you code and travis-ci to build and test. Hope this helps. :)

    jeffa

    L-LL-L--L-LL-L--L-LL-L--
    -R--R-RR-R--R-RR-R--R-RR
    B--B--B--B--B--B--B--B--
    H---H---H---H---H---H---
    (the triplet paradiddle with high-hat)
    
Re: Testing in Perl
by Anonymous Monk on Feb 04, 2016 at 10:06 UTC

    One of the things we are looking to do is run automatic tests (e.g. Test::More) for our code. A discussion arose on what should we test automatically....

    What does that mean "automatic"?

      Good question, sorry about the confusion.
      It means having the tests in a file (eg test.t) which you can run via a cronjob or each time you make changes to the code.