bliako has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks,

I am trying to investigate failed tests which occur on some CPAN testers' systems but not in mine.

Without excluding other suggestions, I would like to make the command make test more verbose by not suppressing diag and STDERR output for each test. Right now, running make test I get:

...
t/11-scripts-pod.t .............................. ok
t/12-from-file.t ................................ 
...
Test Summary Report
-------------------
t/12-from-file.t                              (Wstat: 9 Tests: 11 Fail
+ed: 0)
  Non-zero wait status: 9
  Parse errors: No plan found in TAP output

[download]

So, I would like to know what's going in t/12-from-file.t and where exactly it gets KILL'ed. Alas the output is opaque.

I have read here https://stackoverflow.com/a/5307376 that running make test TEST_VERBOSE=1 will do exactly what I want.

The problem is I don't know how to tell CPAN testers to run make test TEST_VERBOSE=1 (and not the default make test).

The furthest I got was to add ANOTHER make test target in the produced Makefile, with increased verbosity flags, to be run after the usual make test via the MY::postamble technique (using ExtUtils::MakeMaker, e.g. see Re: Benchmarks target in Makefile). The caveat is that I am not sure it will go ahead if previous Makefile target has failed. I think it does not.

Is there a solution to this?

bw, bliako

Replies are listed 'Best First'.
Re: Increase verbosity of "make test"
by Haarg (Priest) on Oct 11, 2023 at 13:17 UTC

    The problem you are trying to diagnose is due to a core perl bug in versions before 5.14. This is triggered by passing a tainted string to YAML::PP::Load, which is causing an infinite loop. On the test machines, it is eventually killed, which gives the status of 9 (SIGKILL).

    Since YAML::PP is parsing the string, not using it in any unsafe way, you should be able to untaint the string before passing it to YAML::PP to avoid the bug. I've filed an issue on YAML::PP about this.

[reply]

      Thanks Haarg++ I would never be able to discover this. Great!

[reply]

      In said module (Data::Roundtrip) I have one instance of feeding YAML::PP with a string. As per your suggestion I am now untainting the string thusly:

      sub yaml2perl {
  my $yaml_string = $_[0];
  ...
  ($yaml_string) = each %{{$yaml_string,0}};
  my $pv = eval { YAML::PP::Load($yaml_string) };
  ...
  return $pv
}

      [download]

      I was previously using YAML but it failed for an (extremely) corner case with a quotes-inside-quotes string. And so I went for YAML::PP as I did not want to use YAML::XS out of concern for users in not dev-friendly environments. YAML::PP does not have a problem with aforementioned corner case.

      I had another choice of restricting use of Data::Roundtrip for Perls >= 5.14. But I keep this as last resort.

      Any opinions welcome.

      Thank Haarg again for looking into this, bw, bliako

[reply]
[d/l]
        I did not want to use YAML::XS out of concern for users in not dev-friendly environments ... Any opinions welcome

        Well, since you are soliciting opinions it would be remiss not to mention that I've found YAML::XS to be significantly faster for parsing to the point where I have modified old, third-party code to use it in preference.

        It depends on your use case and that of other users of your code but you could consider employing one of the many options for conditional dependencies in this scenario rather than explicitly using the PP module.

        For clarity, YAML::PP and YAML::XS are not 100% API compatible so there may be a little work involved in supporting both, should you choose to do so.

        🦛

[reply]
Re: Increase verbosity of "make test"
by Corion (Patriarch) on Oct 11, 2023 at 07:13 UTC

    Output via diag() does not get suppressed by make test (as produced for me):

    # Testing HTTP::Request::FromCurl 0.52, Perl 5.024000
# Algorithm::Loops 1.032
# AutoLoader 5.74
# Carp 1.50
...
# warnings::register 1.04
t/00-load.t .................. ok
# Testing with curl version '7.52.1'

    [download]

    I think output from STDERR does not get suppressed in general, but I'm less certain about that.

[reply]
[d/l]
[select]

      Ouch, Corion you are right. Plus STDERR also is printed. And warn from test file or module is also printed. false alarm sorry

[reply]
[d/l]
Re: Increase verbosity of "make test"
by choroba (Cardinal) on Oct 11, 2023 at 07:13 UTC
    What I do in such a case when a sudden test failure appears in a single environment that I'm not able to reproduce is to contact the tester. I'd ask them gently providing the command to run. Many of the testers are willing to help, even if they don't react promptly.

    Before doing so, though, make sure you've examined the report in detail. The versions of dependencies are usually listed there which might help you reproduce the problem.

    map{substr$_->[0],$_->[1]||0,1}[\*||{},3],[[]],[ref qr-1,-,-1],[{}],[sub{}^*ARGV,3]
[reply]
[d/l]

      yep, CPAN testers were always very helpful to me +++++

[reply]

Back to Seekers of Perl Wisdom