I do a lot of work managing a test suite. It can be quite slow at times and we'd like to capture more useful statistics. If you use the new Test::Harness, you'll find that you can run prove with the --state=save option and it saves the state of the last test run. It doesn't (yet) have much of a public API, but one thing I use it for is figuring out what my slowest tests are. For example, my last run generated the following results:

$ analyze_tests.pl Number of test programs: 47 Total runtime approximately 19 minutes 56 seconds Ten slowest tests: +---------+--------------------------------------------+ | Time | Test | +---------+--------------------------------------------+ | 12m 22s | t/acceptance.t | | 4m 26s | t/aggregate.t | | 0m 31s | t/standards/use.t | | 0m 14s | t/system/both/import/log/pager.t | | 0m 12s | t/system/both/import/log/search.t | | 0m 10s | t/unit/piptest/pprove/testdb.t | | 0m 9s | t/system/both/import/log/log.t | | 0m 8s | t/unit/db/migrations.t | | 0m 7s | t/system/api/v1/programme/programmes.t | | 0m 6s | t/system/both/reports/imports/validation.t | +---------+--------------------------------------------+

Now I can tell at a glance which tests are consuming the most time, but I'd like to add more to this.

First, here's the code I use for the above:

#!/usr/bin/env perl use strict; use warnings; use App::Prove::State; use List::Util 'sum'; use Lingua::EN::Numbers 'num2en'; use Text::Table; sub minutes_and_seconds { my $seconds = shift; return ( int($seconds / 60), int($seconds % 60) ); } my $state = App::Prove::State->new({ store => '.prove' }); my $generation = $state->{_}{generation}; my $tests = $state->{_}{tests}; my $total = sum(map { $_->{elapsed} } values %$tests); my ( $minutes, $seconds ) = minutes_and_seconds($total); my $num_tests = shift || 10; my $total_tests = keys %$tests; if ($num_tests > $total_tests) { $num_tests = $total_tests; } my $num_word = num2en($num_tests); my %time_for; while (my ($test, $data) = each %$tests) { $time_for{$test} = $data->{elapsed}; } my @sorted_by_time_desc = sort { $time_for{$b} <=> $time_for{$a} } keys %time_for; print "Number of test programs: $total_tests\n"; print "Total runtime approximately $minutes minutes $seconds seconds\n +\n"; print "\u$num_word slowest tests:\n"; my $table = Text::Table->new(\'| ', 'Time', \' | ', 'Test', \' |' ); $table->rule(qw(- +)); $table->body_rule(qw(- +)); my @rows; for ( 0 .. $num_tests - 1 ) { my $test = $sorted_by_time_desc[$_]; my $time = $time_for{$test}; my ( $minutes, $seconds ) = minutes_and_seconds($time); push @rows => [ "${minutes}m ${seconds}s", $test, ]; } $table->load(@rows); my @body = map { $table->body($_) } 0 .. $num_tests - 1; print $table->rule(qw(- + )), $table->title, $table->rule(qw(- + )), @body, $table->rule(qw(- + ));

Aside from the lack of public API, we have two main problems:

  1. We don't capture enough information (e.g., number of tests run)
  2. We only save the state for the last test run

If I (or someone who wants a fun project) writes App::Prove::State::SQLite, we can solve these problems. The first would be providing a proper API for App::Prove::State and capturing more information and the second would be persisting information about test runs over time.

The problem with persisting test runs, though, is largely in renaming tests. We'd potentially lose a bit of historical information, but we could still capture the bulk of the info and do things like warn if a particular test suddenly starts taking twice as long to run, or hooking it into Devel::CoverX::Covered to see what covers what and what our coverage percentage is over time.

If anyone has any useful suggestions (or would like to write this and needs someone familiar with the code base), speak up!

Cheers,
Ovid

New address of my CGI Course.


In reply to Managing Test Suites by Ovid

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.