I do a lot of work managing a test suite. It can be quite slow at times and we'd like to capture more useful statistics. If you use the new Test::Harness, you'll find that you can run prove with the --state=save option and it saves the state of the last test run. It doesn't (yet) have much of a public API, but one thing I use it for is figuring out what my slowest tests are. For example, my last run generated the following results:

$ analyze_tests.pl Number of test programs: 47 Total runtime approximately 19 minutes 56 seconds Ten slowest tests: +---------+--------------------------------------------+ | Time | Test | +---------+--------------------------------------------+ | 12m 22s | t/acceptance.t | | 4m 26s | t/aggregate.t | | 0m 31s | t/standards/use.t | | 0m 14s | t/system/both/import/log/pager.t | | 0m 12s | t/system/both/import/log/search.t | | 0m 10s | t/unit/piptest/pprove/testdb.t | | 0m 9s | t/system/both/import/log/log.t | | 0m 8s | t/unit/db/migrations.t | | 0m 7s | t/system/api/v1/programme/programmes.t | | 0m 6s | t/system/both/reports/imports/validation.t | +---------+--------------------------------------------+

Now I can tell at a glance which tests are consuming the most time, but I'd like to add more to this.

First, here's the code I use for the above:

#!/usr/bin/env perl use strict; use warnings; use App::Prove::State; use List::Util 'sum'; use Lingua::EN::Numbers 'num2en'; use Text::Table; sub minutes_and_seconds { my $seconds = shift; return ( int($seconds / 60), int($seconds % 60) ); } my $state = App::Prove::State->new({ store => '.prove' }); my $generation = $state->{_}{generation}; my $tests = $state->{_}{tests}; my $total = sum(map { $_->{elapsed} } values %$tests); my ( $minutes, $seconds ) = minutes_and_seconds($total); my $num_tests = shift || 10; my $total_tests = keys %$tests; if ($num_tests > $total_tests) { $num_tests = $total_tests; } my $num_word = num2en($num_tests); my %time_for; while (my ($test, $data) = each %$tests) { $time_for{$test} = $data->{elapsed}; } my @sorted_by_time_desc = sort { $time_for{$b} <=> $time_for{$a} } keys %time_for; print "Number of test programs: $total_tests\n"; print "Total runtime approximately $minutes minutes $seconds seconds\n +\n"; print "\u$num_word slowest tests:\n"; my $table = Text::Table->new(\'| ', 'Time', \' | ', 'Test', \' |' ); $table->rule(qw(- +)); $table->body_rule(qw(- +)); my @rows; for ( 0 .. $num_tests - 1 ) { my $test = $sorted_by_time_desc[$_]; my $time = $time_for{$test}; my ( $minutes, $seconds ) = minutes_and_seconds($time); push @rows => [ "${minutes}m ${seconds}s", $test, ]; } $table->load(@rows); my @body = map { $table->body($_) } 0 .. $num_tests - 1; print $table->rule(qw(- + )), $table->title, $table->rule(qw(- + )), @body, $table->rule(qw(- + ));

Aside from the lack of public API, we have two main problems:

  1. We don't capture enough information (e.g., number of tests run)
  2. We only save the state for the last test run

If I (or someone who wants a fun project) writes App::Prove::State::SQLite, we can solve these problems. The first would be providing a proper API for App::Prove::State and capturing more information and the second would be persisting information about test runs over time.

The problem with persisting test runs, though, is largely in renaming tests. We'd potentially lose a bit of historical information, but we could still capture the bulk of the info and do things like warn if a particular test suddenly starts taking twice as long to run, or hooking it into Devel::CoverX::Covered to see what covers what and what our coverage percentage is over time.

If anyone has any useful suggestions (or would like to write this and needs someone familiar with the code base), speak up!

Cheers,
Ovid

New address of my CGI Course.

Replies are listed 'Best First'.
Re: Managing Test Suites
by dragonchild (Archbishop) on Apr 28, 2008 at 17:04 UTC
    Why use SQLite when you could use DBM::Deep? That way, you don't have to change anything and it just persists. Writing a SQL generator over it would be pretty simple. We just need something that can work with arrays of hashes and a simple way of indexing. I had started work with this in Presto, but it kinda died when I added transactions to DBM::Deep.

    Oh, and DBM::Deep works everywhere Perl does and doesn't require a compiler.


    My criteria for good software:
    1. Does it work?
    2. Can someone else come in, make a change, and be reasonably certain no bugs were introduced?