in reply to Testing my tests (mutation testing)

If I'm understanding correctly, you're wanting something that can introspect the tests themselves, and make sure that you're asserting on the actual return value of a sub or not. Correct?

I think that may be pretty involved to do, as after some quick thought, you'd need to introspect the test file itself, map a test with the sub in question, and validate whether *all* return paths are asserted against:

# module

sub perform {
    my ($x, $y) = @_;

    if ($x < 10){
        return $x + $y;
    }
    if ($x == 10){
        return $x * $y;
    }
  
    return $x - $y;
}

[download]

Then:

    is perform(5, 5),  10, "...";
    is perform(10, 5), 50, "...";
    is perform(20, 5), 15, "...";

[download]

So it would almost seem as though you'd need to actually write tests against your tests. If there is an existing solution for these types of things, I'd definitely be interested in knowing about it as well.

Replies are listed 'Best First'.
Re^2: Testing my tests
by choroba (Cardinal) on Feb 26, 2017 at 16:09 UTC
    Boolean conditions are handled by Devel::Cover , it tells you whether you tested the TRUE and FALSE branches of them.

    ($q=q:Sq=~/;[c](.)(.)/;chr(-||-|5+lengthSq)`"S|oS2"`map{chr |+ord
}map{substrSq`S_+|`|}3E|-|`7**2-3:)=~y+S|`+$1,++print+eval$q,q,a,

    [download]
[reply]
[d/l]

      Right, but I think what he's asking about is whether the return values themselves have been tested against directly. I just put in the if statements to show that to do what he wants, some serious thought would have to be put into the introspection to ensure that if you could prove that an assertion has been made, it would also have to be confirmed which return path the return came out of.

[reply]

        "Right, but I think what he's asking about is whether the return values themselves have been tested against directly."

        Don't assume, ASK! But consider that if you test your interfaces then those return values WILL BE IN THE TEST. That's what a test does -- it checks that the return value is what you expected. Coverage shows you which branches have been executed by those tests and which have not.

        Testing is simple. People make it complicated.

[reply]
Re^2: Testing my tests
by BrowserUk (Patriarch) on Feb 26, 2017 at 18:42 UTC
    So it would almost seem as though you'd need to actually write tests against your tests.

    Won't he then need some tests to test the tests he uses to test his tests?

[reply]

      and then a guard to guard the tests against change.

      Premature optimization is the root of all job security
[reply]

        I've just always felt that if I have 100% coverage of each sub, I must be testing the return value. To boot, I always overlap tests so that it's often not just one test file that uses the return of a sub, so that if I break that return, I may end up with broken tests in several places, if not just several tests within a single file.

[reply]
Re^2: Testing my tests
by szabgab (Priest) on Feb 26, 2017 at 16:17 UTC
    No. I would like to introspect and change my module or application and then run the unchanged tests again. (Updated the original post too with this clarification.)
[reply]

      What about running the tests, then modifying the sub, then running again? Here's a quickly thrown together example (in this example I simply mock out the whole function) to see if this is more along the lines of what you're looking for:

      package Package; {

    sub perform {
        return $_[0] + $_[1];
    }
}

package main; {
    use Mock::Sub;
    use Test::More;

    tests();

    my $m = Mock::Sub->new;
    my $changed = $m->mock('Package::perform');
    $changed->return_value($_[0] - $_[1]);

    tests();

    done_testing();

    sub tests {
        is Package::perform(5, 5), 10, "5 + 5 = 10";
    } 
}

      [download]

      Output:

      ok 1 - 5 + 5 = 10
not ok 2 - 5 + 5 = 10
#   Failed test '5 + 5 = 10'
#   at pack.pl line 24.
#          got: '0'
#     expected: '10'
1..2
# Looks like you failed 1 test of 2.

      [download]

      If this example is more along the lines you're after (modifying code on the fly), my Devel::Examine::Subs is designed to alter code within a file (so say you wanted to modify a single line in a single sub within a package, you could (then revert it back), and I could write a mock up example of what it may look like. But perhaps I'm way off here.

[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom