Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical
 
PerlMonks  

Test::Harness bug ? ... or author idiocy ?

by syphilis (Archbishop)
on Nov 05, 2021 at 11:09 UTC ( [id://11138456]=perlquestion: print w/replies, xml ) Need Help??

syphilis has asked for the wisdom of the Perl Monks concerning the following question:

Hi,

There's an issue I've struck a few times with perl extensions that I've authored, where I've put a debug printf() statement inside an XSub in the XS file.
It's always a totally banal printf() statement - eg printf("In the 'if' block\n"); or printf("In the else block\n"):.
It's pretty simple stuff, serves its purpose well, and is generally useful.

However, if:
1) I then run make test on this perl extension;
&&
2) in the running of a particular test script, this printf() statement gets called tens of thousands of times
then make test will report that the test script has failed, though all tests have passed.
I guess it's something to do with the way that the Test::Harness suppresses the printf() statements.
(If I change the printf("...\n") to warn("...\n") then, IIRC, there's no such problem.)

I'm not sure that I've ever struck this behaviour on a non-Windows system.

Is this a bug that should be reported ? ... or merely a behaviour that no street savvy author would ever admit having encountered ?

Cheers,
Rob

Replies are listed 'Best First'.
Re: Test::Harness bug ? ... or author idiocy ?
by NERDVANA (Deacon) on Nov 05, 2021 at 15:16 UTC
    print goes to STDOUT, and warn goes to STDERR. The text on STDOUT is parsed as TAP. So if one of your many printf() happened to print text to STDOUT that could be interpreted as unsuccessful TAP (or just inject content into the stream of valid TAP that breaks its syntax), then this could be a legitimate failure. Afaik, STDERR is always seen as logging data and never parsed as TAP. Anything your script writes to STDOUT should be prefixed with a "# " to ensure it can't be parsed as TAP. Don't forget if your printf wraps lines, each line would need to begin with "# ". In general you should just remove any printing to STDOUT in your XS code. Is there any reason to use printf instead of warn for diagnostics?
      Is there any reason to use printf instead of warn for diagnostics?

      If I use warn then it gets output to the screen every time it gets executed in make test - creating a deluge of noise to sift through.
      If I use printf then I can quietly fiddle about with pieces of code and see what's going on, and then not be slugged with the deluge if I run make test.
      To that extent, it's working perfectly ... the only problem is that TAP::Harness likes to give me misleading messages that I don't really understand.

      It's just a practice I engage in as a convenience thing during debugging, and such printf statements are not something that I'll (intentionally ;-) include in a CPAN release.
      I don't regard this TAP::Harness behaviour as a big issue ... but I'm a little curious about it.

      Cheers,
      Rob
Re: Test::Harness bug ? ... or author idiocy ?
by hippo (Bishop) on Nov 05, 2021 at 11:17 UTC
    Is this a bug that should be reported ?

    I would say yes. It is at the very least unexpected behaviour and should therefore be documented if not fixed outright. It might well be a symptom of some deeper as-yet-unknown problem so the Test::Harness maintainers should definitely be informed.

    If you can supply a reproducible test case then no doubt others could test and determine if it is indeed an MSWin32-only issue.


    🦛

      If you can supply a reproducible test case then no doubt others could test and determine if it is indeed an MSWin32-only issue.

      Actually I can (and just have) tested it out on Ubuntu, perl-5.34.0 and it's the same issue.

      Here's an Inline::C demo of the problem:
      Firstly, with the printf() statement commented out:
      use strict; use warnings; use Inline C => <<'EOC'; SV * foo( SV * in ) { /* printf("# %s\n", SvPV_nolen(in)); */ return newSVpv(SvPV_nolen(in), 0); } EOC my $ok = 1; print "1..1\n"; for(1 .. 1000) { my $n = int(rand(10000000)); $ok = 0 if foo("$n") ne "$n"; } if($ok) { print "ok 1\n" } else { print "not ok 1\n"; } __END__
      I first run perl try.pl to compile the script, and then I run:
      $ perl -MTAP::Harness -e '$h = TAP::Harness->new(); $h->runtests("./tr +y.pl");' ./try.pl .. ok All tests successful. Files=1, Tests=1, 0 wallclock secs ( 0.04 usr 0.01 sys + 0.06 cusr + 0.01 csys = 0.12 CPU) Result: PASS
      As you can see, everything is fine. But if I then include the printf() statement and run perl try.pl (to recompile the script), upon re-testing I get:
      $ perl -MTAP::Harness -e '$h = TAP::Harness->new(); $h->runtests("./t +ry.pl");' ./try.pl .. All 1 subtests passed Test Summary Report ------------------- ./try.pl (Wstat: 0 Tests: 1 Failed: 0) Parse errors: No plan found in TAP output Files=1, Tests=1, 1 wallclock secs ( 0.11 usr 0.01 sys + 0.05 cusr + 0.02 csys = 0.19 CPU) Result: FAIL
      Cheers,
      Rob

        Thanks for this example. I have run your code with the printf uncommented and reproduced your findings (on Linux) but only on some runs. The result is inconsistent. Here is the output from 2 runs within a few seconds of each other:

        10045 $ perl -MTAP::Harness -e '$h = TAP::Harness->new(); $h->runtests +("./11138492.pl");' ./11138492.pl .. ok All tests successful. Files=1, Tests=1, 0 wallclock secs ( 0.06 usr 0.00 sys + 0.04 cusr + 0.00 csys = 0.10 CPU) Result: PASS 10046 $ perl -MTAP::Harness -e '$h = TAP::Harness->new(); $h->runtests +("./11138492.pl");' ./11138492.pl .. All 1 subtests passed Test Summary Report ------------------- ./11138492.pl (Wstat: 0 Tests: 1 Failed: 0) Parse errors: No plan found in TAP output Files=1, Tests=1, 0 wallclock secs ( 0.06 usr 0.01 sys + 0.04 cusr + 0.00 csys = 0.11 CPU) Result: FAIL

        However, upon inspection the problem is not with TAP::Harness. The output of just running the script shows that the plan can appear in the middle of your data. Again, running a few times gives:

        10046 $ perl 11138492.pl | grep -Fn 1..1 830:# 79678951..1 10047 $ perl 11138492.pl | grep -Fn 1..1 830:# 691..1 10047 $ perl 11138492.pl | grep -Fn 1..1 829:# 5056771..1

        So this looks to be a buffering issue. If I simply modify your code to include $|++; before the plan is printed then the script runs fine, the plan is printed at the top and the test harness has no complaints.

        Not a bug in Test::Harness after all.


        🦛

Re: Test::Harness bug ? ... or author idiocy ? (#TAP)
by Anonymous Monk on Nov 05, 2021 at 12:26 UTC
    printf("# this is TAP comment" )

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://11138456]
Approved by philipbailey
Front-paged by davies
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others rifling through the Monastery: (5)
As of 2024-04-19 07:30 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found