Some tests require user input. When you're testing one by one that isn't a big deal. When you're repeatedly testing all of your modules, that is going to get very annoying, very quickly.

At least some thought needs to be put into this. Even if it is as simple as, "Make it possible to uninstall an annoying test."

I've discovered this problem in my attempt to test all of the CPAN. I wound up using $SIG{ALRM} and simply skip tests where I run into this issue. It's not perfect, but it's a workable hack for me. Is this portable?

Cheers,
Ovid

New address of my CGI Course.

In Vanilla/Strawberry Perl builds, I found that setting $ENV{PERL_MM_USE_DEFAULT} and $ENV{AUTOMATED_TESTING} to true cuts out many of those annoying prompts for user input.

-xdg

Code written by xdg and posted on PerlMonks is public domain. It is provided as is with no warranties, express or implied, of any kind. Posted code may not have been tested. Use of posted code is at your own risk.

I'd expect this to be portable, but I wouldn't be surprised if some tests still prompt for user input in a way that this doesn't catch.

Interesting idea. Here's my suggestion for the issues you raised:

1. The makefile for the module must be installed along with the test files in order to re-run tests, and
2. yes, the output must be captured and stored somewhere -
3. so I'd suggest (similar to what AutoSplit does for AutoLoader - store baz.al for e.g. Foo::Bar::Quux in auto/Foo/Bar/Quux/baz.al) to install them in test/Foo/Bar/Quux/t/{00_use.t,01_foo.t,...}. Makefile and logs from test runs could live there, also.

Since the test is a separate tree, it doesn't mess with the lib tree.

The Makefile could live in test/Foo/Bar/Quux, the logs could be written to test/Foo/Bar/Quux/logs.

What do you think?

--shmem

_($_=" "x(1<<5)."?\n".q·/)Oo.  G°\        /
                              /\_¯/(q    /
----------------------------  \__(m.====·.(_("always off the crowd"))."·
");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}

4. What assumptions are we making about how tests are packaged?

Scenarios:

• test.pl vs t/*.t
• Custom Makefile.PL or Build.PL that affects test runs
• build_requires modules bundled in inc/

I'm not convinced that you can get this idea to work short of caching the full distribution directory or tarball at install-time and then iterating through those using the actual Makefile.PL or Build.PL files to prep and call tests.

Considering that something has to set @INC to run the tests against uninstalled modules, is it as simple as installing the modules, then running the tests without setting @INC?

Install them where? And how do I track them? And how can I easily run them again, know where they are, ensure not have my previously accepted failures get re-reported as failures (I'd have to track failures on tests for code which I'm installing), etc. There are plenty of subtle issues which make this more difficult than it first seems. I suspect many power users have installed packages for which some tests have failed.

Cheers,
Ovid

New address of my CGI Course.

Install them where?

Install what where? Modules? That part really ought to work. If it doesn't, run the tests against the installed modules and see if they fail. Note that this part of the building and installation process requires extra work to set @INC appropriately. If you're building and installing a module, you can run the tests again, avoid doing that extra work, and let them test the just-installed modules.

If you mean something different by them, you've really confused me and I really don't understand what you're saying. I may not anyway.

What problem are you trying to solve by installing tests? When will you run the installed tests? What information will running installed tests give you? Why is that information valuable?

Randy Sims was kind enough to forward me Ovid's post to the Perl-QA list on this topic, here's my response:

Yes, I've been thinking about this for a long time. In fact, in the most recent M::B beta I made some steps toward it, by adding a 'retest' action that's just like 'test' except that it doesn't look in blib/, just in @INC. Functionally that actually covers a lot of the same ground you're after.

What I like about the 'retest' approach is that it's very easy and it's much more likely to work. It also makes it possible to run old tests against a new installation, or vice versa. What I don't like about it is that the user has to find the tarball again that they previously installed. I can imagine that in some situations that wouldn't be trivial. In other situations, when people can plan ahead, it's probably not a big deal.

I think there are some larger issues than 1,2,3 above that you might have missed, too:

4) Many distributions, including many of the most crucial and well-used ones, have some extra set-up steps in their build/install sequences. Others make assumptions in their test suites about where various files are located relative to the test code or relative to the current working directory. It's quite possible that in order for "installed" tests to work correctly it could take some serious coöperation by modules' authors.

5) Where should tests be installed? Where would any other supporting materials be installed?

For your #1 above, I'd say just perform a reinstall. For #2, maybe just punt - is there much of a need for that? For #3, I think we can work it into Module::Build as an action or flag(s) to the 'install' action. For EU::MM-based modules I'm not sure what the best approach would be, but probably I don't have to think about it. =)

-Ken

retest sounds nice; it would let you run the tests for shiny new Foo-1.23 against the installed Foo-1.2 before deciding to install the new version. (Yes, this has issues if 1.23 is a feature release, not just a bugfix release.)

With a retest action in place, I'm not sure there's a need for installing the tests; just keep the module build directory intact and run retest as desired. I like to do that anyway, with {config,make,test,install}.log files saved in it to preserve record of any problems during the original install.

The problem is that right now, we have no such thing as module tests. We have only distribution tests. A huge number of test suites contain a subset of tests that assume they are being run on an unpacked module distribution.

The Module::Build retest action mentioned above sounds like a good way of setting up an environment that respects this assumption for testing installed modules. That's at least a good first step...

I did a lot more thinking about this and wrote more about a possible solution in my use.perl journal.

---

Once it's Turing complete, everything else is just syntactic sugar.

BTW, I think it's unfortunate that ActivePerl's PPM packages don't include tests. It's not because it tested OK on the platform they built the distribution on, that it'll be fine on the computer you want to work on. There could be external DLLs that are missing, or different, or there could be platform differences between for example WinXP and Win98. Take Win32::NetResource for example, there's no way it can run on Win98, because the DLL it depends on, is just a stub on Win98.