Portably unit testing scripts

wanna_code_perl has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks!

I have a script in my distribution that allows convenient CLI access to my library. It's handy, but it's not the main focus of my distribution. Still, I would like to get unit test coverage for it if I can.

The basic skeleton of my script is below, but it is built around the "modulino" approach of main(@ARGV) unless caller suggested by blogs such as this one (Gabor Szabo). How can I add unit tests for things like:

Test that --help and --version output reasonable text and exiting with the correct value.
Test that main() does the right thing if @ARGV is not empty
Test that main() accepts input from standard input if @ARGV is empty.

Test::Output would get me part of the way there, but anything that actually exits, such as pod2usage, exits the entire test script, which causes a failure. I could actually execute the script, but that isn't much of a "unit" test anymore and also defeats the purpose of the "modulino" style of unit testing scripts like this, and doing that portably would be tricky.

(Portability is a key requirement here.)

All of the searching I've done seems to stop at "shove everything into a sub, and unit test that", but I've not been able to find any useful examples that actually dive into the specifics of common tasks like pod2usage, anything that calls exit for any reason, scripts that read from standard input, etc.

My script is handy and worthy of distribution, but it's still just a thin wrapper around my library. If I were to skip unit tests for those things, my script would have barely any useful unit tests, and then I might be inclined to not include it in the distribution at all.

How would you unit test something like this? I also obviously do my own integration testing before every release, but that serves a different purpose.

#!/usr/bin/env perl

use 5.010;
use strict;
use warnings;

my $VERSION = '1.00';
use Getopt::Long 'GetOptionsFromArray';
use Pod::Usage;

main(@ARGV) unless caller;

sub main {
    my $ARGV = \@_;

    my %args = (
        help    => sub { pod2usage -verbose => 1, -exitval => 0 },
        version => sub { say "my_script version $VERSION\n"; exit },
    );
    GetOptionsFromArray($ARGV, \%args, qw<host=s port=i password=s
        help version> ) or pod2usage(2);

    if (@ARGV) {
        do_something($_) for @ARGV;
    } else {
        while (<STDIN>) { chomp; do_something($_) }
    }
}

sub do_something {
    say "do_something: $_[0]";
}
[download]

Comment on Portably unit testing scripts Select or Download Code

Replies are listed 'Best First'.
Re: Portably unit testing scripts by haukex (Archbishop) on Oct 26, 2019 at 07:46 UTC
I could actually execute the script, but that isn't much of a "unit" test anymore Why not? As you said, it's a thin wrapper around your library, and I assume you've got plenty of tests for that, so you've got that covered. The script is an interface between the command line and the library, and that's the "unit" you're testing here - IMHO actually executing Perl is the only "real" way to test it. In any case, don't be too strict with yourself regarding the formal definitions of "unit" and "integration" tests here - while certainly important in some contexts (e.g. large projects with multiple devs), IMHO the far more important metrics here are whether you're testing everything, as in code coverage and so on. I have various command-line tools that I test, see e.g. `t/` in htools, although in that particular repository I haven't tested them on Windows. For that, you could take a look at the tests for my Badge::Simple command-line tool, or perhaps even more closely related to your question, this test for my Shell::Tools. Here's a short example for your script, assuming it's in a `bin` subdir and the tests in a `t` subdir, and if the script actually had a Synopsis POD. Running prove on this works on Linux and on Windows. #!/usr/bin/env perl use warnings; use strict; use FindBin; use File::Spec::Functions qw/catfile updir/; use Capture::Tiny 'capture'; use Config; use Test::More tests => 2; # Unit Under Test my $UUT = catfile($FindBin::Bin, updir, 'bin', 'foo.pl'); subtest '--version' => sub { plan tests => 3; my ($out,$err,$exit) = capture { system $Config{perlpath}, $UUT, '--version' }; is $exit, 0, 'perl ok'; is $out, "my_script version 1.00\n\n", 'stdout message correct'; is $err, '', 'no stderr'; }; subtest '--help' => sub { plan tests => 3; my ($out,$err,$exit) = capture { system $Config{perlpath}, $UUT, '--help' }; is $exit, 0, 'perl ok'; like $out, qr/\bUsage:\s+foo\.pl FOO BAR\b/si, 'stdout message correct'; is $err, '', 'no stderr'; }; [download] (Note: I've often used $^X instead of Config's `perlpath` without problems, but the CPAN Authors FAQ recommends the latter.)	[reply] [d/l] [select]
Re^2: Portably unit testing scripts by wanna_code_perl (Friar) on Oct 26, 2019 at 09:39 UTC
IMHO the far more important metrics here are whether you're testing everything, as in code coverage and so on. Yes, you seem to understand what I'm trying to do: I want to test everything adequately (and part of that is code coverage). Portability is a very important concern. In fact, my main reservation about using `$^X`, `system`, `Capture::Tiny`, etc, (which is very similar to what I used to do for in-house releases for known platforms) is a concern, potentially unfounded, that I might artificially limit the platforms my distribution will install on. On the one hand, maybe I am worried for nothing, but on the other hand, I'd hate for my entire module installation to fail because a test for the script did not succeed, when the library itself would have tested and installed just fine. Thanks for the link to your distributions that use a similar approach. Those CPAN testers matrices give me hope that I might be worried for no good reason. I know there is more I could do, but a lesser desire is that I keep my finite efforts focused on the library itself, with proportionately less time spent on some 40-line script I am including as a useful afterthought. But, hey, that's software sometimes, I suppose.	[reply] [d/l] [select]
Re^3: Portably unit testing scripts by haukex (Archbishop) on Oct 26, 2019 at 10:26 UTC
I might artificially limit the platforms my distribution will install on Yeah, I like portability myself, and I had the same worry when I started releasing CPAN modules, but experience has shown me that portability is possible. And even if you release a module and it initially fails on a bunch of CPAN Testers, it's possible to bring it back to 100%. As you probably saw, the Shell::Tools tests (which my example code above is based on) pass on Cygwin, Darwin (Mac OS X), FreeBSD, GNU/kFreeBSD, Haiku, Linux, MSWin32, NetBSD, OpenBSD, and Solaris.	[reply]
Re^2: Portably unit testing scripts by Anonymous Monk on Oct 26, 2019 at 08:34 UTC
The thing about $^X is shocking / unconvincing / inadequately documented. In a test suite $^X is always absolute.	[reply]
Re^3: Portably unit testing scripts by Anonymous Monk on Oct 26, 2019 at 08:52 UTC
uh, what?	[reply]
Re: Portably unit testing scripts by Discipulus (Canon) on Oct 26, 2019 at 12:11 UTC
Hello wanna_code_perl, First I second what haukex said. If you have a library, a perl module, test what this module exports. The cli is just an interface. Second: your example is not a modulino. A modulino is a perl module `.pm` file that can be invoked as a program via a trick. A modulino should end with the famous `1;` or cannot be loaded. If your is a modulino you can `use` it in your testing program and play with their subs, testing them as you wish. In both cases, modulino or standalone script, you can test it somehow, using Capture::Tiny (without embarking yourself in a fight with Run3 modules.. ;). you can also launch the script and inspect the exit values. Few example I have in some test on github something you can adapt: use strict; use warnings; use Test::More; use Test::Exception; use Capture::Tiny qw(capture); ... # expected death my ($out, $err, @res) = capture { dies_ok { a_sub_that_dies() } 'expected to die without argumen +ts'; }; # verbosity check ($out, $err, @res) = capture { $obj->method( verbose => 2 ); }; ok ((split "\n", $out) > 30 , "30+ lines expected with verbosity = 2") +; # checking exit status (you can check 'perl your_modulino_or_script ar +g1 arg2..' ) my $exit = system "$program /? >nul 2>&1"; ok ( 16 == ($exit>>8), "[$program /? >nul 2>&1] returned the expected +value"); [download] See also testing in my bibliotheca and Test::Script from CPAN. L* There are no rules, there are no thumbs.. Reinvent the wheel, then learn The Wheel; may be one day you reinvent one of THE WHEELS.	[reply] [d/l] [select]
Re: Portably unit testing scripts by jcb (Parson) on Oct 25, 2019 at 23:55 UTC
Could you stash the actual `CORE::exit` coderef somewhere and try overriding `CORE::GLOBAL::exit`? Or simply import an `exit` sub into the module's namespace, which will override `exit` in that namespace only. (You may need to establish this import before loading the module; I am uncertain about this.) To test `pod2usage` in that, you will probably need to place your own `exit` mock at `Pod::Usage::exit`, and it should probably use a non-local exit, such as `die` or `last`, instead of returning normally.	[reply] [d/l] [select]
Re: Portably unit testing scripts by stevieb (Canon) on Oct 26, 2019 at 22:26 UTC
Your script is a front-end to the module you've written. You could move argument parsing into the module so that the binary (ie. script) only ever makes function/method calls to the module; the script does only its own basic logic. Then, the binary is irrelevant and you need not worry about testing it at all, less some compile checks. `pod2usage` and other 'system' calls have their own tests written upon install, so you can almost certainly skip these. If you're that concerned, run a test to dump this output somewhere, then have a different, later test run POD and layout checks against the output of that sample file. Anything that calls `exit()` can be caught with, like jcb said, a mock, or other forms of trickery (such as having a test file dump the output of a system call to an in-memory file, then check that). Note that you can run a script under system() and get actual return values. Again, unless your script is a core part of your distribution (sounds like it isn't), make it as simple as possible. I might want to write my own because yours is too simple. Then, I can either use your publicized `options()` function/method, write my own for my own script, or utilize both yours append my own. In the end, you don't really want to test the script anyways (where possible). You want to test what the script represents. The more you incorporate into the module(s), the less worry you have about testing the front-end script(s).	[reply] [d/l] [select]
Re: Portably unit testing scripts (no modulinos) by Anonymous Monk on Oct 26, 2019 at 07:31 UTC
The basic skeleton of my script is below, but it is built around the "modulino" approach of main(@ARGV) unless caller .... How would you unit test something like this? I could actually execute the script, but that isn't much of a "unit" test anymore and also defeats the purpose of the "modulino" style of unit testing scripts like this, and doing that portably would be tricky. Hi, What purpose does that modulino stuff serve anyway? The most it can accomplish is to to substitute 1 symlink for 1 regular file -- thats silly golf , esp in a distribution pod2usage exitval does accept "noexit"	[reply]
Re^2: Portably unit testing scripts (no modulinos) by wanna_code_perl (Friar) on Oct 26, 2019 at 09:17 UTC
What purpose does that modulino stuff serve anyway? The idea is to make unit testing of applications easier/possible, since the main control flow gets put into a subroutine that can be unit tested like any other sub. The most it can accomplish is to to substitute 1 symlink for 1 regular file -- thats silly golf , esp in a distribution What? It's not designed to shorten (golf) anything... in fact, the code is a bit more verbose. And it has nothing to do with the filesystem. pod2usage exitval does accept "noexit" That's right, "NOEXIT" would help with testing the `--help / pod2usage()` path. Unfortunately that doesn't help the general case, though.	[reply] [d/l]
Re^3: Portably unit testing scripts (no modulinos) by Anonymous Monk on Oct 26, 2019 at 09:32 UTC
Having a main sub and all code in subs is plain modularity. The modulino part is unless caller. Deciding if act as a module or program based on caller is silly modulino golf.	[reply]