almr has asked for the wisdom of the Perl Monks concerning the following question:

Hi, I'm using "prove" to run tests produced by shell scripts. These shell scripts should be tested with various shells. So, manually, prove -e sh -j9; prove -e bash -j9 etc.

Now, I'd like to run the tests in parallel for all the shells (that's where the bottleneck really is). I can see several options:

  1. Create a bunch of wrapper scripts that run the actual tests, and test the Nt * Nsh wrappers instead of the actual tests. Horrible.
  2. Set up the harness manually and call runtests with multiple aliases for each .t file
  3. Extend TAP::Harness
  4. Learn more about Plugin, SourceHandler

Where should I look?

Replies are listed 'Best First'.
Re: prove: parallel multiple shells
by Corion (Patriarch) on Aug 18, 2021 at 08:16 UTC

    WWW::Mechanize::Shell runs the same test suite with different Chrome versions. For that, I wrote another wrapper that runs the test suite using prove in parallel for each Chrome version.

    When the tests are not run through that wrapper, each test runs its suite using all Chrome versions found. For that I wrote a helper module to run each test suite in a loop, which is likely not what you want:

    use lib 't/';
use helper;

# What instances of Chrome will we try?
my @instances = t::helper::browser_instances();

if (my $err = t::helper::default_unavailable) {
    plan skip_all => "Couldn't connect to Chrome: $@";
    exit
} else {
    plan tests => 4*@instances;
};

sub new_mech {
    t::helper::need_minimum_chrome_version( '62.0.0.0', @_ );
    WWW::Mechanize::Chrome->new(
        autodie => 1,
        @_,
    );
};

t::helper::run_across_instances(\@instances, \&new_mech, 4, sub {
    my ($browser_instance, $mech) = @_;
    ...
});

    [download]
[reply]
[d/l]
[select]

      Hey, cool. To my shame I can't quite figure out were the parallelization takes place... I see a NestedLoops call in WWW-Mechanize-Chrome, so that's sequential. Is it in system("$Config{ make } test"), which presumably calls prove... which looks for .t files?

      Or is it in t::helper::run_across_instances call, in the snippet above?

[reply]

        Sorry - no, I leave all the parallelization still to prove. It runs all test files in parallel (well, up to a limit). prove is called by make test, as you correctly surmised.

        The run_across_instances just loops (single-tasked) through the Chrome instances.

[reply]
[d/l]
[select]
Re: prove: parallel multiple shells
by haukex (Archbishop) on Aug 18, 2021 at 08:00 UTC

    Interesting question, I'm not sure at the moment what the best approach might be - could you perhaps provide a minimal example of what the actual code being run looks like so we can play around with it? (SSCCE) Also, what machines are these tests intended to be run on - only ones under your control, or on users' machines as well? (i.e. how portable does it have to be)

[reply]

      It's t3st (gitlab.com: kstr0k/t3st -- can't post links I think), a shell testing library that I'm working on. It aims for... infinite scripting flexibility, or something. There are self-tests (in t/). The tests are shell scripts that produce TAP (using the library), and can run either via their own shebang, or via prove's "-e" option.

      It currently runs under a lot of shells (multiple bash versions, zsh emulations etc), and it's not a biggie because the tests are simple; however I'd like to offer users (or at least myself in other projects) the option to parallelize different shells for potentially complex tests.

[reply]
[d/l]
[reply]

        Thanks for that info! Out of curiosity, are you an active/keen Perl user? Or just using TAP for testing your t3st Lightweight, flexible shell TAP testing library?

        After visiting testanything.org for the first time just now, it seems to me that Perl's humble little Test Anything Protocol of 1987 is more popular than ever! Going from strength to strength. In terms of programming history, I assume TAP was invented by Larry Wall in 1987 (testanything.org history does not say). Can anyone confirm? Oh, and any cool references or anecdotes on the history of TAP (especially from the early years) are especially welcome.

        Update: the testanything.org history page refers to "Sam Villan’s historical Perl GIT repository" (without providing a link). Anyone know what that is? I wonder if it's a typo and they mean Sam Vilain?

[reply]

Back to Seekers of Perl Wisdom