Automate and dispatch unit test runs across perl/berrybrew on remote Windows and Unix systems

My Test::BrewBuild test deployment system is now reasonably stable, and has the ability to dispatch test runs to remote test servers. This means with a single command, you can run your unit tests across any number of perl instances on any Operating Systems automatically, and get the results delivered back to you.

To do this, we use the bbdispatch to send brewbuild commands to previously configured remote testers. To start a bbtester, log on to the system and run bbtester start. That'll start the tester and put it in the background.

Now's probably a good time to state that Git is required to be installed on all systems used for network testing, and one should peruse the basic system setup doc.

In these examples, I have three testers set up. tester1 is the DNS name of a Ubuntu Linux system running perlbrew, tester2 is a Windows 2008 Server running berrybrew, and the localhost is a FreeBSD server, again running perlbrew.

There are three flags for bbdispatch:

-c: A quoted string containing the brewbuild commands to run. Eg: "brewbuild -i 5.20.3 -R". If omitted, we will default to simply "brewbuild"
-r: A string containing a repository you want to test. Eg: https://github.com/stevieb9/mock-sub. If you are currently in a repository directory, we'll automatically fetch and use it and you can omit this flag
-t: These are your testers. This flag can be specified multiple times.

Note that you can alternatively use a config file to store the dispatcher information.

Here's the most basic example. We're already in a repository working directory so we can omit -r, we're only working on a tester on localhost, and we'll just use the default brewbuild test run:

# start a tester on the localhost

me@host:~/repos/mock-sub$ bbtester start

Started the Test::BrewBuild test server at PID 20584 on IP address 0.0
+.0.0 and TCP port 7800...

# dispatch a test run of my [metamod://Mock::Sub] 
# module (from within the repo dir)

me@host:~/repos/mock-sub$ bbdispatch -t localhost

localhost - x86_64-linux

5.18.4 :: PASS
5.20.3 :: PASS
5.22.1 :: PASS
[download]

Things get more useful when you have multiple testers across multiple different Operating Systems.

This example does a basic run using the same repo as above, but this time I'm explicitly setting it. I'm also dispatching to three tester systems:

$ bbdispatch -t localhost -t tester1 -t tester2 -r https://github.com/
+stevieb9/mock-sub

tester1 - MSWin32-x64-multi-thread

5.22.1_64 :: PASS
5.10.1_32 :: PASS
5.20.3_32 :: PASS

tester2 - x86_64-linux

5.18.4 :: PASS
5.20.3 :: PASS
5.22.1 :: PASS

localhost - amd64-freebsd

5.10.1 :: PASS
5.14.4 :: PASS
5.22.1 :: PASS
[download]

Below, we use the -c flag to tell brewbuild that we want to perform test on the current module (I'm back in the repo dir again so I omit -r), and then have brewbuild run unit tests of all the module's reverse dependencies (-R) to ensure our proposed updated module doesn't break down-river (ie. modules that require/use your module) modules.

$ bbdispatch -t localhost -t tester1 -t tester2 -c "brewbuild -R"

tester1 - MSWin32-x64-multi-thread

reverse dependencies: File::Edit::Portable, Devel::Examine::Subs, Deve
+l::Trace::Subs

File::Edit::Portable
5.10.1_32 :: PASS
5.20.3_32 :: PASS
5.22.1_64 :: PASS

Devel::Examine::Subs
5.20.3_32 :: PASS
5.22.1_64 :: PASS
5.10.1_32 :: FAIL

Devel::Trace::Subs
5.20.3_32 :: PASS
5.22.1_64 :: PASS
5.10.1_32 :: FAIL


localhost - amd64-freebsd

reverse dependencies: File::Edit::Portable, Devel::Examine::Subs, Deve
+l::Trace::Subs

File::Edit::Portable
5.10.1 :: PASS
5.22.1 :: PASS
5.14.4 :: FAIL

Devel::Examine::Subs
5.10.1 :: PASS
5.14.4 :: PASS
5.22.1 :: PASS

Devel::Trace::Subs
5.10.1 :: PASS
5.14.4 :: PASS
5.22.1 :: FAIL


tester2 - x86_64-linux

reverse dependencies: File::Edit::Portable, Devel::Examine::Subs, Deve
+l::Trace::Subs

File::Edit::Portable
5.18.4 :: PASS
5.20.3 :: PASS
5.22.1 :: PASS

Devel::Examine::Subs
5.18.4 :: PASS
5.20.3 :: PASS
5.22.1 :: PASS

Devel::Trace::Subs
5.18.4 :: PASS
5.20.3 :: PASS
5.22.1 :: FAIL
[download]

Notice that some results are FAIL. In this case, we create a bblog directory in the directory you're working in, and generate log files for each individual fail. This allows you to see what broke and where, without having to go to each individual system. You can then update/fix code, then run another dispatch. Here's an example of the files that were generated by the above run:

$ ls bblog

tester1_Devel-Examine-Subs-5.10.1_32-FAIL.bblog
tester1_Devel-Trace-Subs-5.10.1_32-FAIL.bblog
tester2_Devel-Trace-Subs-5.22.1-FAIL.bblog
localhost_Devel-Trace-Subs-5.22.1-FAIL.bblog
localhost_File-Edit-Portable-5.14.4-FAIL.bblog
[download]

The log files contain all errors that the tester would have produced to STDOUT and STDERR, with the cpanm build logs appended.

On a normal dispatch run (without running revdep), the log file would have appeared as tester1_5.10.1_32-FAIL.bblog. Running brewbuild in standalone mode (no dispatching): 5.10.1_32-FAIL.bblog. You can optionally save even the PASS logs if you choose by using the -S, --save flag to brewbuild.