simonm has asked for the wisdom of the Perl Monks concerning the following question:

Greetings, fellow monks.

I seek your wisdom regarding the organization of test scripts in large CPAN distributions. Specifically, Class::MakeMethods has numerous subclasses, each with several tests, and I'm trying to arrange the resulting pile of over 100 test scripts.

My intent has been to group the tests so that the more basic tests ran first, with new and experimental modules tested later, and so that related tests would run together.

In older releases, I added a digit to the start of each test's name to control its position in the test sequence. This works, although it seems a bit opaque, and it scales poorly as new tests are added.

Subsequently, I tried breaking the distribution up into smaller chunks (1, 2, 3), but this made distribution and updates more of a hassle. When recombining the distributions, I experimented with packaging them together, with a primary makefile that recursed down to subdirectories each with their own makefile and collection of tests. However, some light reading seemed to suggest that this might not be very portable.

Most recently, I tried using a single makefile, but kept the tests in subdirectories of t/, and passed 'TESTS' => 't/*/*.t' to MakeMaker's WriteMakefile. Unfortunately, that fails due to limitations of glob() on some platforms (read, Windows).

The one thing that I know will work is to go back to one big directory with numeric prefixes, but I'm not quite ready to give up. The machines I have easy access to are all Unix variants, so I hoped to find some cross-platform assistance here.

Is there some simple, portable way of accomplishing this multi-level directory globbing so I can keep my tests in subdirectories without overburdening my Makefile.PL? (File::DosGlob seems to handle directory recursion; would conditionally including it on Windows do me any good?)

At first glance, it doesn't seem like switching from MakeMaker to Module::Build would make any difference; am I overlooking something?

Alternately, is there some existing set of practices that would help to address this?

Many thanks in advance for any suggestions.

  • Comment on MakeMaker and multiple test directories

Replies are listed 'Best First'.
Re: MakeMaker and multiple test directories
by PodMaster (Abbot) on Jul 21, 2003 at 22:10 UTC
    ExtUtils::MakeMakers WriteMakefile function takes 
        test => { TESTS => 't/*.t' },

    [download]
    as a parameter. You can set it to whatever, and the resulting makefile executes something like 
    C:\Perl\bin\perl.exe "-MExtUtils::Command::MM" "-e" "test_harness(0, '
+blib\lib', 'blib\arch')" t\001_load.t ...

    [download]
    which may or may not agree with various shells (which have their own limits). I suggest you do kinda like Mail::Box does, and have a simple test.pl, one which uses Test::Harness to run the various test files in any way you please. Actually, something like the following is better: 
    #!/usr/bin/perl
# file: test.pl

use ExtUtils::Command::MM;
@ARGV = qw[
    mytests/01-load-This.t
    ...
];
test_harness( $ENV{TEST_VERBOSE} || 0, 'blib\lib', 'blib\arch');

    [download]
    Combine that with test => { TESTS => 'test.pl' }, and you've got gold ;)

    MJD says "you can't just make shit up and expect the computer to know what you mean, retardo!"
    I run a Win32 PPM repository for perl 5.6.x and 5.8.x -- I take requests (README).
    ** The third rule of perl club is a statement of fact: pod is sexy.

[reply]
[d/l]
[select]

      Unfortunately, it's pyrite. CPAN ignores the return value of test.pl, so it will silently install modules for which the tests fail. You might as well not distribute tests in that case.

[reply]
[d/l]
        Really? If that is true, it is still just a failing of CPAN, nothing less, nothing more. It does not invalidate the tests in any way (the tests are not there to stop cpan from installing a module).

        MJD says "you can't just make shit up and expect the computer to know what you mean, retardo!"
        I run a Win32 PPM repository for perl 5.6.x and 5.8.x -- I take requests (README).
        ** The third rule of perl club is a statement of fact: pod is sexy.

[reply]
Re: MakeMaker and multiple test directories
by chromatic (Archbishop) on Jul 21, 2003 at 21:56 UTC

    Here's the guts of the Module::Build extension I use in Jellybean.

    sub ACTION_test
{
    my $self = shift;
    my @files;
    find( sub { /\.t\z/ and push @files, $File::Find::name }, 't' );
    $self->SUPER::ACTION_test( 'test_files=' . join(' ', @files ) );
}

    [download]

    There's no reason you couldn't do something similar in your Makefile.PL, though you might run into command-line length limits.

[reply]
[d/l]
[select]
Re: MakeMaker and multiple test directories
by adrianh (Chancellor) on Jul 27, 2003 at 14:31 UTC
    Alternately, is there some existing set of practices that would help to address this?

    An alternate approach is not to have lots of *.t files :-)

    I tend to have a single test script for each class (sometimes based on several test modules - often based on Test::Class).

    If you move 100s of *.t files into a much smaller number of files you can also see a quite respectable increase in the speed of running your tests - since you avoid the overhead of starting perl and compiling your modules.

[reply]
Re: MakeMaker and multiple test directories
by BrowserUk (Patriarch) on Jul 21, 2003 at 23:23 UTC

    Unfortunately, that fails due to limitations of glob() on some platforms (read, Windows).

    Could you clarify on the limitation. Perl's standard glob function seems to work fine for me? 

    P:\test>perl -le" print for glob 'c:/test/words/*/*';
c:/test/words/a/aa
c:/test/words/a/ab
c:/test/words/a/ac
c:/test/words/a/ah
c:/test/words/a/ak
c:/test/words/a/al
c:/test/words/a/am
c:/test/words/a/an
c:/test/words/a/ar
c:/test/words/a/as
c:/test/words/a/au
c:/test/words/a/aw
...
c:/test/words/c/aa
c:/test/words/c/ab
c:/test/words/c/ac
c:/test/words/c/ad
c:/test/words/c/ae
c:/test/words/c/af
c:/test/words/c/ag
...

    [download]
    Examine what is said, not who speaks.
    "Efficiency is intelligent laziness." -David Dunham
    "When I'm working on a problem, I never think about beauty. I think only how to solve the problem. But when I have finished, if the solution is not beautiful, I know it is wrong." -Richard Buckminster Fuller
[reply]
[d/l]

      Perl's standard glob function seems to work fine for me?

      Looking at the bug report again, it seems I may have misinterpreted it -- it looks like the limitation is not with the Perl glob() function. Instead, the problem is that MakeMaker ends up passing the the pattern to the test script on the command line, such that it's the Windows command shell that's responsible for the wildcard matching, not Perl's glob().

      Good catch, and I'll update my original post. (If I can figure out how to do so...)

[reply]
Re: MakeMaker and multiple test directories
by simonm (Vicar) on Sep 15, 2003 at 18:38 UTC
    For the record, the way I ended up tackling this was to create a t/foo.t test that ran all of the sub-tests in t/foo/*.t. This isn't a perfect solution, but I think it's enough for now.
[reply]
      Ah-ha!

      It turns out that Regexp::Common has a fix for this in its Makefile.PL: 

      my @tests = qw {t/*.t t/*/*.t};
WriteMakefile (
    ...
    test         => {TESTS => $^O eq 'MSWin32'  
                              ? "@{[map {glob} @tests]}" : "@tests" }
);

      [download]

      That rocks! I plan to incorporate it in the next release of Class::MakeMethods.

[reply]
[d/l]
[reply]

Back to Seekers of Perl Wisdom