prove --state=failed,save

1nickt has asked for the wisdom of the Perl Monks concerning the following question:

I am trying to use prove --state=failed,save. But it seems to not work as documented.

The first thing that is unclear is whether "test" in the doc refers to individual tests, or to test files. So I tried it both ways.

At the urging of Bishop toolic I am posting here my test files and results.

With multiple tests in a single file:

$ rm .prove

$ cat t/test.t
use Test::More tests => 4;
ok 1 == 1, 'A';
ok 1 == 2, 'B';
ok 1 == 1, 'C';
ok 1 == 2, 'D';

$ prove -v --state=save t/test.t
t/test.t ..
1..4
ok 1 - A
not ok 2 - B

#   Failed test 'B'
#   at t/test.t line 3.
ok 3 - C
not ok 4 - D

#   Failed test 'D'
#   at t/test.t line 5.
# Looks like you failed 2 tests of 4.
Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/4 subtests

Test Summary Report
-------------------
t/test.t (Wstat: 512 Tests: 4 Failed: 2)
  Failed tests:  2, 4
  Non-zero exit status: 2
Files=1, Tests=4,  0 wallclock secs ( 0.02 usr  0.01 sys +  0.03 cusr 
+ 0.00 csys =  0.06 CPU)
Result: FAIL

$ cat .prove
---
generation: 1
last_run_time: 1519583552.87203
tests:
  t/test.t:
    elapsed: 0.0406849384307861
    gen: 1
    last_fail_time: 1519583552.87122
    last_result: 3
    last_run_time: 1519583552.87122
    last_todo: 0
    seq: 1
    total_failures: 1
version: 1
...

$ prove -v --state=failed,save t/test.t
t/test.t ..
t/test.t ..
1..4
ok 1 - A
not ok 2 - B

#   Failed test 'B'
#   at t/test.t line 3.
ok 3 - C
not ok 4 - D

#   Failed test 'D'
#   at t/test.t line 5.
# Looks like you failed 2 tests of 4.
Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/4 subtests

Test Summary Report
-------------------
t/test.t (Wstat: 512 Tests: 4 Failed: 2)
  Failed tests:  2, 4
  Non-zero exit status: 2
Files=1, Tests=4,  0 wallclock secs ( 0.02 usr  0.01 sys +  0.02 cusr 
+ 0.00 csys =  0.05 CPU)
Result: FAIL

$ cat .prove
---
generation: 2
last_run_time: 1519583756.96688
tests:
  t/test1.t:
    elapsed: 0.0349948406219482
    gen: 2
    last_fail_time: 1519583756.96619
    last_result: 3
    last_run_time: 1519583756.96619
    last_todo: 0
    mtime: 1519583524
    seq: 2
    total_failures: 2
version: 1
...
[download]

As you can see all four tests were run the second time as well as the first time. There also does not seem to be anything in the .prove file that records the outcome of the individual tests.

With multiple test files:

As you can see, again all four test files were run the second time as well as the first time. The only difference is that on the second run the failing test files were run first, and then the ones that had succeeded on the first run. Again, this is not the behaviour the documentation describes.

Any clues gratefully accepted, especially if you are using this feature yourself with success.

The way forward always starts with a minimal test.

Comment on prove --state=failed,save Select or Download Code

Replies are listed 'Best First'.
Re: prove --state=failed,save by choroba (Cardinal) on Feb 25, 2018 at 20:27 UTC
--state operates on files, not individual tests, as they can depend on each other (even if it's a bad practice we should avoid). If you only want to run the failed tests, don't tell prove to run anything else: $ prove --state=failed t/02.t .. 1/1 # Failed test 'B' # at t/02.t line 2. # Looks like you failed 1 test of 1. t/02.t .. Dubious, test returned 1 (wstat 256, 0x100) Failed 1/1 subtests t/04.t .. 1/1 # Failed test 'D' # at t/04.t line 2. # Looks like you failed 1 test of 1. t/04.t .. Dubious, test returned 1 (wstat 256, 0x100) Failed 1/1 subtests Test Summary Report ------------------- t/02.t (Wstat: 256 Tests: 1 Failed: 1) Failed test: 1 Non-zero exit status: 1 t/04.t (Wstat: 256 Tests: 1 Failed: 1) Failed test: 1 Non-zero exit status: 1 Files=2, Tests=2, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.08 cusr + 0.00 csys = 0.10 CPU) Result: FAIL [download] By appending `t` to the command line, you told `prove` to run all the tests in `t/` together with the failed ones. ($q=q:Sq=~/;[c](.)(.)/;chr(-\|\|-\|5+lengthSq)`"S\|oS2"`map{chr \|+ord }map{substrSq`S_+\|`\|}3E\|-\|`7**2-3:)=~y+S\|`+$1,++print+eval$q,q,a, [download]	[reply] [d/l] [select]
Re^2: prove --state=failed,save by 1nickt (Canon) on Feb 25, 2018 at 23:35 UTC
Thank you choroba, Bravo!! (you went from "choroba has never used --state" to figuring the answer out rather quickly ;-) Update: Although it seems to be not quite true that "By appending t to the command line, you told prove to run all* the tests in t/ together with the failed ones."* ... apparently `prove` skips from the directory any test files that have already been run via `failed`. Since I never rely on `prove`'s looking for `t/` by default it did not occur to me that the examples in the doc are showing complete commands. I think `--state` would be more useful for me if it applied to individual tests within files (and within subtests even!) Maybe a plugin is called for. The way forward always starts with a minimal test.	[reply] [d/l] [select]