How do track down underlying reason for smoke test failure

nysus has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: How do track down underlying reason for smoke test failure by marto (Cardinal) on Sep 12, 2019 at 09:38 UTC
"Do I set up a perl distro with the same exact module config..." Since your module is on github you can add Continuous_integration via Travis_CI with very little effort, and no financial cost, have it test your code against several versions of perl, on different OS etc. Create your .travis.yml (or see Travis-CI Helpers for Perl), setup Travis CI, activate the repos and it'll do it automatically from then on.	[reply]
Re: How do track down underlying reason for smoke test failure by bliako (Abbot) on Sep 12, 2019 at 10:35 UTC
I have managed to reproduce the problem (Perl 5.26.2, fedora linux) when I omitted `./` from INC dir - as `Test::Classifier` requires `t::TestMods::Test::Processor` : `prove -Iblib/lib -It/TestMods t/01-file-collector.t` [download] t/01-file-collector.t .. # Running my tests t/01-file-collector.t .. 1/13 # Failed test 'creates an object' # at t/01-file-collector.t line 30. # died: Test::Classifier is not a Role::Tiny at /home/andreas/Download +s/File-Collector-0.037/blib/lib/File/Collector.pm line 391. # Failed test 'has files property' # at t/01-file-collector.t line 34. # Failed test 'has common_dir property' # at t/01-file-collector.t line 38. Can't call method "get_count" on unblessed reference at t/01-file-coll +ector.t line 41. # Looks like your test exited with 2 just after 4. [download] This works fine on my system: `prove -Iblib/lib -It/TestMods -I. t/01-file-collector.t` [download] Could it be that some of the modules in `t/TestMods` do not load correctly under certain circumstances? A comment on your code if I may (`File-Collector-0.037 / lib / File / Collector.pm`): `# eval class code foreach my $class ( @$classes ) { eval "require $class"; }` [download] Adding a check on `eval` will tell you if `require`ed class was not found or failed to load. e.g. from eval, e.g. `eval "xyz"; die $@ if $@;` I make this comment because the test fails when constructor sets `@classes` to contain `Test::Classifier`. Does this module load OK? Same goes for `AUTOLOAD` in said file: `my $obj = $class->new($s->{_files}{all}, \($s->{selected}), $s->{_files}{"${cat}_files"}{_files}); return $obj;` [download] You may want to check whether `$obj` materialised before returning instead of relying on caller checking if got undef. bw, bliako	[reply] [d/l] [select]
Re^2: How do track down underlying reason for smoke test failure by nysus (Parson) on Sep 12, 2019 at 11:16 UTC
Nice find. Thanks. That led me to see that the failed tester has `PERL_USE_UNSAFE_INC = 0` in the reports. In one of the reports that passed, it is set to "1". I'm not completely clear why it fails, though. My test file has `use lib 't/TestMods';` in it. So I would think the `File::Collector` would be able to find my module in there. I don't quite follow how the non-existence of '.' in @INC causes the test to fail. $PM = "Perl Monk's"; $MCF = "Most Clueless ~~Friar~~ ~~Abbot~~ ~~Bishop~~ ~~Pontiff~~ ~~Deacon~~ ~~Curate~~ ~~Priest~~ Vicar"; $nysus = $PM . ' ' . $MCF; Click here if you love Perl Monks	[reply] [d/l] [select]
Re^3: How do track down underlying reason for smoke test failure by bliako (Abbot) on Sep 12, 2019 at 11:27 UTC
I'm not completely clear why it fails, though. My test file has use lib 't/TestMods'; in it. So I would think the File::Collector would be able to find my module in there. I don't quite follow how existence of '.' in @INC plays into this. If my theory is right, loading `Test::Classifier` fails under certain INC. Because requiring these: `use t::TestMods::Test::Processor; use t::TestMods::Test::TestObj;` [download] in `t/TestMods/Test/Classifier.pm`, requires an INC of `./` bw, bliako edit: changed pre to blockquote	[reply] [d/l] [select]
Re^4: How do track down underlying reason for smoke test failure by nysus (Parson) on Sep 12, 2019 at 11:49 UTC
Re^4: How do track down underlying reason for smoke test failure by nysus (Parson) on Sep 12, 2019 at 11:30 UTC
Re^5: How do track down underlying reason for smoke test failure by syphilis (Archbishop) on Sep 12, 2019 at 12:36 UTC
Re^5: How do track down underlying reason for smoke test failure by bliako (Abbot) on Sep 12, 2019 at 12:24 UTC
Some notes below your chosen depth have not been shown here
Re: How do track down underlying reason for smoke test failure by davido (Cardinal) on Sep 12, 2019 at 17:25 UTC
You have the ability to add environment exploration to your tests, of course. Your tests run on the smoker. So add a `diag` to your tests that produces output about some of the hunches you want to chase down -- things that don't already show up in the smoker's output but that you think would be helpful to know. Also when your tests encounter failures, there will be an entry in http://analysis.cpantesters.org/ for your module. Sometimes it takes 24 hours or more to show up, and it only shows up when the distribution has test failures. As a hint, you can create a development version number -- see perlmodstyle for how to do that, and that will allow you to iterate without impacting your typical users. Dave	[reply] [d/l]