prove --state=failed,save

1nickt has asked for the wisdom of the Perl Monks concerning the following question:

I am trying to use prove --state=failed,save. But it seems to not work as documented.

The first thing that is unclear is whether "test" in the doc refers to individual tests, or to test files. So I tried it both ways.

At the urging of Bishop toolic I am posting here my test files and results.

With multiple tests in a single file:

As you can see all four tests were run the second time as well as the first time. There also does not seem to be anything in the .prove file that records the outcome of the individual tests.

With multiple test files:

$ rm .prove

$ cat t/test1.t
use Test::More tests => 1;
ok 1 == 1, 'A';

$ cat t/test2.t
use Test::More tests => 1;
ok 1 == 2, 'B';

$ cat t/test3.t
use Test::More tests => 1;
ok 1 == 1, 'C';

$ cat t/test4.t
use Test::More tests => 1;
ok 1 == 2, 'D';

$ prove -v --state=save t
t/test1.t ..
1..1
ok 1 - A
ok
t/test2.t ..
1..1
not ok 1 - B

#   Failed test 'B'
#   at t/test2.t line 2.
# Looks like you failed 1 test of 1.
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1 subtests
t/test3.t ..
1..1
ok 1 - C
ok
t/test4.t ..
1..1
not ok 1 - D

#   Failed test 'D'
#   at t/test4.t line 2.
# Looks like you failed 1 test of 1.
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1 subtests

Test Summary Report
-------------------
t/test2.t (Wstat: 256 Tests: 1 Failed: 1)
  Failed test:  1
  Non-zero exit status: 1
t/test4.t (Wstat: 256 Tests: 1 Failed: 1)
  Failed test:  1
  Non-zero exit status: 1
Files=4, Tests=4,  0 wallclock secs ( 0.03 usr  0.02 sys +  0.12 cusr 
+ 0.02 csys =  0.19 CPU)
Result: FAIL

$ cat .prove
---
generation: 1
last_run_time: 1519584062.28557
tests:
  t/test1.t:
    elapsed: 0.0381917953491211
    gen: 1
    last_pass_time: 1519584062.14733
    last_result: 0
    last_run_time: 1519584062.14733
    last_todo: 0
    seq: 1
    total_passes: 1
  t/test2.t:
    elapsed: 0.0359680652618408
    gen: 1
    last_fail_time: 1519584062.1927
    last_result: 2
    last_run_time: 1519584062.1927
    last_todo: 0
    seq: 2
    total_failures: 1
  t/test3.t:
    elapsed: 0.0373549461364746
    gen: 1
    last_pass_time: 1519584062.23968
    last_result: 0
    last_run_time: 1519584062.23968
    last_todo: 0
    seq: 3
    total_passes: 1
  t/test4.t:
    elapsed: 0.0356581211090088
    gen: 1
    last_fail_time: 1519584062.28471
    last_result: 2
    last_run_time: 1519584062.28471
    last_todo: 0
    seq: 4
    total_failures: 1
version: 1
...

$ prove -v --state=failed,save t
t/test2.t ..
1..1
not ok 1 - B

#   Failed test 'B'
#   at t/test2.t line 2.
# Looks like you failed 1 test of 1.
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1 subtests
t/test4.t ..
1..1
not ok 1 - D

#   Failed test 'D'
#   at t/test4.t line 2.
# Looks like you failed 1 test of 1.
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1 subtests
t/test1.t ..
1..1
ok 1 - A
ok
t/test3.t ..
1..1
ok 1 - C
ok

Test Summary Report
-------------------
t/test2.t (Wstat: 256 Tests: 1 Failed: 1)
  Failed test:  1
  Non-zero exit status: 1
t/test4.t (Wstat: 256 Tests: 1 Failed: 1)
  Failed test:  1
  Non-zero exit status: 1
Files=4, Tests=4,  0 wallclock secs ( 0.02 usr  0.01 sys +  0.12 cusr 
+ 0.02 csys =  0.17 CPU)
Result: FAIL

$ cat .prove
---
generation: 2
last_run_time: 1519584153.98762
tests:
  t/test1.t:
    elapsed: 0.0381758213043213
    gen: 2
    last_pass_time: 1519584153.94307
    last_result: 0
    last_run_time: 1519584153.94307
    last_todo: 0
    mtime: 1519583913
    seq: 7
    total_passes: 2
  t/test2.t:
    elapsed: 0.0387568473815918
    gen: 2
    last_fail_time: 1519584153.84872
    last_result: 2
    last_run_time: 1519584153.84872
    last_todo: 0
    mtime: 1519583940
    seq: 5
    total_failures: 2
  t/test3.t:
    elapsed: 0.0351169109344482
    gen: 2
    last_pass_time: 1519584153.98684
    last_result: 0
    last_run_time: 1519584153.98684
    last_todo: 0
    mtime: 1519583962
    seq: 8
    total_passes: 2
  t/test4.t:
    elapsed: 0.0388789176940918
    gen: 2
    last_fail_time: 1519584153.89626
    last_result: 2
    last_run_time: 1519584153.89626
    last_todo: 0
    mtime: 1519583950
    seq: 6
    total_failures: 2
version: 1
...
[download]

As you can see, again all four test files were run the second time as well as the first time. The only difference is that on the second run the failing test files were run first, and then the ones that had succeeded on the first run. Again, this is not the behaviour the documentation describes.

Any clues gratefully accepted, especially if you are using this feature yourself with success.

The way forward always starts with a minimal test.

Comment on prove --state=failed,save Select or Download Code

Replies are listed 'Best First'.
Re: prove --state=failed,save by choroba (Cardinal) on Feb 25, 2018 at 20:27 UTC
--state operates on files, not individual tests, as they can depend on each other (even if it's a bad practice we should avoid). If you only want to run the failed tests, don't tell prove to run anything else: $ prove --state=failed t/02.t .. 1/1 # Failed test 'B' # at t/02.t line 2. # Looks like you failed 1 test of 1. t/02.t .. Dubious, test returned 1 (wstat 256, 0x100) Failed 1/1 subtests t/04.t .. 1/1 # Failed test 'D' # at t/04.t line 2. # Looks like you failed 1 test of 1. t/04.t .. Dubious, test returned 1 (wstat 256, 0x100) Failed 1/1 subtests Test Summary Report ------------------- t/02.t (Wstat: 256 Tests: 1 Failed: 1) Failed test: 1 Non-zero exit status: 1 t/04.t (Wstat: 256 Tests: 1 Failed: 1) Failed test: 1 Non-zero exit status: 1 Files=2, Tests=2, 0 wallclock secs ( 0.02 usr 0.00 sys + 0.08 cusr + 0.00 csys = 0.10 CPU) Result: FAIL [download] By appending `t` to the command line, you told `prove` to run all the tests in `t/` together with the failed ones. ($q=q:Sq=~/;[c](.)(.)/;chr(-\|\|-\|5+lengthSq)`"S\|oS2"`map{chr \|+ord }map{substrSq`S_+\|`\|}3E\|-\|`7**2-3:)=~y+S\|`+$1,++print+eval$q,q,a, [download]	[reply] [d/l] [select]
Re^2: prove --state=failed,save by 1nickt (Canon) on Feb 25, 2018 at 23:35 UTC
Thank you choroba, Bravo!! (you went from "choroba has never used --state" to figuring the answer out rather quickly ;-) Update: Although it seems to be not quite true that "By appending t to the command line, you told prove to run all* the tests in t/ together with the failed ones."* ... apparently `prove` skips from the directory any test files that have already been run via `failed`. Since I never rely on `prove`'s looking for `t/` by default it did not occur to me that the examples in the doc are showing complete commands. I think `--state` would be more useful for me if it applied to individual tests within files (and within subtests even!) Maybe a plugin is called for. The way forward always starts with a minimal test.	[reply] [d/l] [select]