Devel::Cover can easily show if a certain function or expression was executed during the test run, but it cannot tell if there was an assertion checking the validity of the result.

One could randomly change the code under test (e.g. replace a + by a - ) and run the tests again. If they still pass, we have a problem. The tests do not check that code properly.

Is there a tool for Perl that would automate this process?

For further clarification, I'd like to change the source code of the module or application under test and leave my tests unchanged.

Update: Use Case

To further elaborate assume you have a huge code-base with a huge test suite. You pick a module that has 100% coverage and wonder can I safely refactor this? Will the test suite catch if I make a mistake? Lanx gave an excellent and very simple example of code with a problematic test 
sub foo { 
   my ($x, $y) = @_;
   return $x + $y;
}

[download]
The test: 
is foo(2, 0), 2

[download]
How can I estimate the risk of changing this code? One possible way is to change the code in a way that should break it and see if the tests fail. So I'd have a tool that can introspect the source code of my application and change the code at a random place. e.g. it would change the + in the above function to -. (No mocking, really changing the code on the disk.)

The tests would still pass.

This is an indication that the tests don't protect me at that point.

Answer

Apparently this is called "mutation testing" and there are several modules on CPAN that do this:

See details under this node: Re: Testing my tests

In reply to Testing my tests (mutation testing) by szabgab

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.