in reply to Re: Testing my tests
in thread Testing my tests (mutation testing)

"You may want to try generating random input for your unit tests..."

We all do this, by default, don't we? To further, I don't think tests can be (easily) set up to be tested by other tests, but I digress.

A sub should be tested thoroughly so that the return is tested explicitly. This should just be part of the default test regimen. If a new return path is created, it should be added, and tested all the same.

Unless there's some form of AI going on to monitor changes to the sub itself, to see whether the return has changed (or a new return path created), I don't think it's feasible to do what is being asked here.

I suppose a test involving PPI could be used to monitor the structure of the sub itself, but that's going pretty deep (and delves into my comment about hacking live-files live-time above).

Either way, I'd love to see something like this if it's ever presented.

Replies are listed 'Best First'.
Re^3: Testing my tests
by LanX (Saint) on Feb 27, 2017 at 02:54 UTC
    > We all do this, by default, don't we?

    manually yes, not generated.

    Szabgab only hinted about tests that handle code like $a+ switched to $a-

    so in the following case

     
sub foo { 
   my ($a,$b) = @_;
   return $a + $b;
}

    [download]

    testing foo(*,0) (with * any number) will always pass, even if you change + to -

    If several pairs of values were generated by a random generator and the result was approved manually, then the likelyhood of a false positive would be minimal.

    And new test input could be generated quickly, depending on the manual decision if the code change was really intended or just a bug.

    As I said to (semi-) automatize this one needed to parse already a lot of the functions body.

    Anyway I don't want to elaborate more, without further explanation by the OP. (the requirements sound paradoxial)

    Cheers Rolf
    (addicted to the Perl Programming Language and ☆☆☆☆ :)
    Je suis Charlie!

[reply]
[d/l]
[select]
      I am not sure why do you see the requirements paradoxical. You were the closest by far to what I was thinking and brought an excellent example. I've updated the original post with that and some further explanation.

      And you are right this probably requires the full introspection of all the code.

[reply]
        > the requirements paradoxical.

        because without further explanation it sounded like

        • wanting tests that never fail
        • or having test heuristics intelligent enough to make you unemployed.

        at least now I'm sure what your up to.

        Regarding your update, you can't avoid false positive tests.

        But like I said you can reduce the likelyhood of them drastically for a reasonable prize by bundling many random test vectors.

        Some my produce false positives, but hardly all of them.

        In our case (2,0) may be OK but (666,42) would not.

        Cheers Rolf
        (addicted to the Perl Programming Language and ☆☆☆☆ :)
        Je suis Charlie!

[reply]
      "(the requirements sound paradoxial)"

      They do indeed, and with the respect I have for the OP, that's why I've been throwing out such odd concepts all day.

      I don't know what exactly OP is looking for. Gabor didn't come back with further insight, so trial by current knowledge or at best peer-review is all we could throw forward.

[reply]
Re^3: Testing my tests
by RonW (Parson) on Feb 27, 2017 at 23:42 UTC
    I don't think tests can be (easily) set up to be tested by other tests

    To test a test requires replacing what the test is testing with the test-tester.

    For tests that call a function in the app or module being tested, the called function would be replaced a "stub" function that does 2 main things: Check and log the supplied inputs, then return a result. The result could be completely random, randomly chosen from a list or the next in sequence (which would require the test-tester preserve it's current state).

    For tests, like the above, where the tests are "driving" the testing, it will be necessary to run the test-under-test enough times to insure full coverage of the test-under-test.

    For test where the app or module under test is driving the testing, test-tester will be able to perform the test as many ties as needed.

[reply]

In Section Seekers of Perl Wisdom