Automated Edge Testing

thekestrel has asked for the wisdom of the Perl Monks concerning the following question:

Monks,

Following a post I made here on testing Edges (i.e. Not just testing what should be going into a subroutine, but what shouldn't) and I wanted to see if you can created Automated tests as I've just started writing the testing routines in t/ for my module and there is around 20 per subroutine and that is just the basics. I'm not sure if there are ways that exist that already do this, but I wanted to try out my ideas.

I know that good module design says, write the tests and documentation first and then thee may chisel out some code, but I know that this isn't always the case in practice. The amount of times I've typed make test and had it say 1 test successful i.e. There was just the default test created after they used h2xs to make the module.

So I was thinking how could one write some code to auto generate some tests and this is what I came up with. Similar to electric fence when you want to check for memory overruns in C, instead of just hooking Malloc I hooked every single subroutine. Now every subroutine gets routed through my hooked versions first. From there you can analyze the incoming data on the fly to make generalizations about the system.

So the idea is you run this setup script over your module and it hooks all of the subroutines and calls from the subroutines. Then you run all of your scripts that exercise your module and its gathers information about them. After you've finished running tests you run a second script which uses the logged data to generate .t test files that can be used in make test. The neat thing is you know the configuration of 'real' data so you can send the functions sample data an expect the same results or moreover you now know what it isn't expecting generally.

So to that end here is generalize_setup.plx which hooks all of the code. *I'm sure somewhere a whole group of readable code Gods are turning over in their graves (if they died =P), with this code, but this is the first version*

( Update: Now have two files per subroutine .data which has the runtime data and .type which stores the operator type information into a subroutine. It can also handle a module a few layers down i.e. My::Deep::Module. Also cleaned up code a lil' (ok everything is relative =P) )

#!/usr/bin/perl

use strict;
use warnings;

#Check command line arguments
die "Incorrect number of arguments\n\tFormat : generalize_setup <file>
+\n\n" if ( $#ARGV != 0 );

my $file = $ARGV[0];
my ($file_noext, $ext) = split '\.', $file;

#load in the code to hooked
print "Loading file \'$file\' for processing\n";
open "IN", "<$file" or die "Could not open file for read : $!\n";
my @old_code = <IN>;
close(IN);

#Back up the old code
print "Backing up file \'$file\' to \'$file.bak\'\n";
open "BACKUP", ">$file.bak" or die "Could not open file for write : $!
+\n";
print BACKUP @old_code;
close (BACKUP);

my @new_code;
my $linenum = 1;
my $subcount = 0;
my @subroutines;
my $packageline;
my $package;

my $dir = "";
my $new_package = "";

#do the subroutines
foreach (@old_code) {
    if ( $linenum == 2) {
        my $pl = $packageline;
        $pl = s/package/use/;
        push @new_code, "$pl;\n";
    }
    if ( /^package/ ) {
        $packageline = $_;
        $packageline =~ /\w+\s+(.+)\;/;
        $package = $1;
        my @bits = split "::", $package;
        for ( 0 .. scalar @bits - 2 ) {
            $dir .= $bits[$_] . "/"; 
            $new_package .= $bits[$_] . "::"; 
        }
        $new_package .= "_" . $bits[scalar @bits - 1];
        chop $dir;
        push @new_code, "package $new_package;\n";
    } elsif ( /^sub/ ) {
        my ($pre, $subroutine, $post) = split ' ', $_;
        push @new_code, "$pre __$subroutine $post\n";
        $subcount++;
        push @subroutines, $subroutine;
    } else {
        push @new_code, $_;
    }
    $linenum++;
}


#set the data directory
my $datadir = "$dir\\/data";

#do the calls
foreach (@new_code) {
    
    #s/add_input\s*\(/$package\:\:add_input\(/;
    foreach my $s ( @subroutines ) {
        if ( /$s\s*\(/ ) {
            if ( !/>$s/ ) { 
                s/$s\s*\(/$package\:\:$s\(/;
            }
        }
        #fix up &func's 
        if ( /\&$s/ ) {
            s/$s/__$s/;
        }
    }
}

#Write the modified old code 
open "OUT", ">$dir/_Neural.pm" or die "Could not open file for write :
+ $!\n";
print OUT $_ foreach ( @new_code );
close(OUT);

#Create the hooks
print "Replacing \'$file\' code with hooked version\n"; 
open "HOOK", ">$file" or die "Could not open file for write : $!\n";
print HOOK "$packageline\n\n";
print HOOK "use $new_package;\n\n";
print HOOK "use attributes;\n\n";
foreach my $sub ( @subroutines ) {
    print HOOK <<__END__;

#Hook from __$sub to $sub
sub $sub {
    my \$string;
    open "SUB_TYPE", ">>$datadir\\/$sub.type" or die "Could not open f
+ile for write : \$!\\n";
    open "SUB_DATA", ">>$datadir\\/$sub.data" or die "Could not open f
+ile for write : \$!\\n";

    my \$args;
    foreach my \$arg ( \@_ ) {
        \$args .= \$arg . ","; 
    }
    chop \$args;
    \$args .= "\n";
    print SUB_DATA \$args;
    close (SUB_DATA);

    foreach \$attr ( \@_ ) {
        if ( ref(\$attr) ) {
            \$string .= "REF:" . ref(\$attr) . ",";
        } else {
            \$string .= attributes::reftype(\\\$attr) . ",";
        }
    }
    chop \$string;
    \$string .= "\\n";
    print SUB_TYPE \$string;
    close (SUB_TYPE);
    return $new_package\:\:__$sub(\@_);
}

__END__
}
print HOOK "1;\n";
close (HOOK);

#Create a directory and blank file(s) for data
system ("mkdir $datadir");
foreach my $sub ( @subroutines ) {
    open "SUB", ">$datadir/$sub.type" or die "Could not open file for 
+write : $!\n";
    close (SUB);
}

print "Hooked $subcount subroutines\n\nNow exercise code...\n\n";
[download]

Here is an example of the code that is produced for the hooks.

package AI::Neural;


use AI::_Neural;

use attributes;


#Hook from __new to new
sub new {
    my $string;
    open "SUB_TYPE", ">>AI\/data\/new.type" or die "Could not open fil
+e for write : $!\n";
    open "SUB_DATA", ">>AI\/data\/new.data" or die "Could not open fil
+e for write : $!\n";

    my $args;
    foreach my $arg ( @_ ) {
        $args .= $arg . ","; 
    }
    chop $args;
    $args .= "\n";
    print SUB_DATA $args;
    close (SUB_DATA);

    foreach $attr ( @_ ) {
        if ( ref($attr) ) {
            $string .= "REF:" . ref($attr) . ",";
        } else {
            $string .= attributes::reftype(\$attr) . ",";
        }
    }
    chop $string;
    $string .= "\n";
    print SUB_TYPE $string;
    close (SUB_TYPE);
    return AI::_Neural::__new(@_);
}

    .....

    .....
[download]

Running it over my module we get

Loading file 'Neural.pm' for processing
Backing up file 'Neural.pm' to 'Neural.pm.bak'
Replacing 'Neural.pm' code with hooked version
Hooked 37 subroutines

Now exercise code...
[download]

Now run one of my scripts that excercises the module. The script runs as per normal as expected (just a smidgen slower =P. I create a directory called data and store data on each of the subroutines. As you can see only subroutines that get exercised have data. A .type file is created for each subroutine as we can still make tests that check type just not check the data.

total 21
drwxr-xr-x+ 2 paul matthews mkgroup-l-d   0 Apr  1 20:21 .
drwxr-xr-x+ 3 paul matthews mkgroup-l-d   0 Apr  1 20:28 ..
-rw-r--r--  1 paul matthews mkgroup-l-d   0 Apr  1 20:21 AddEdgeOverRa
+ndomVertex.type
-rw-r--r--  1 paul matthews mkgroup-l-d   0 Apr  1 20:21 AddRandomConn
+ectedNeuron.type
-rw-r--r--  1 paul matthews mkgroup-l-d   0 Apr  1 20:21 AddRandomEdge
+.type
-rw-r--r--  1 paul matthews mkgroup-l-d   0 Apr  1 20:21 AddRandomNeur
+on.type
-rw-r--r--  1 paul matthews mkgroup-l-d   0 Apr  1 20:21 AddVertexOver
+RandomEdge.type
-rw-r--r--  1 paul matthews mkgroup-l-d   0 Apr  1 20:21 DeepCopy.type
-rw-r--r--  1 paul matthews mkgroup-l-d 144 Apr  1 20:21 GetNewVertexN
+umber.data
-rw-r--r--  1 paul matthews mkgroup-l-d  11 Apr  1 20:21 GetNewVertexN
+umber.type
-rw-r--r--  1 paul matthews mkgroup-l-d   0 Apr  1 20:21 RandomizeVert
+exActivationFunction.type
-rw-r--r--  1 paul matthews mkgroup-l-d   0 Apr  1 20:21 SaveNetwork.t
+ype
-rw-r--r--  1 paul matthews mkgroup-l-d   0 Apr  1 20:21 XMLout.type
-rw-r--r--  1 paul matthews mkgroup-l-d   6 Apr  1 20:21 __PACKAGE
-rw-r--r--  1 paul matthews mkgroup-l-d   0 Apr  1 20:21 activation_si
+gmoid.type
-rw-r--r--  1 paul matthews mkgroup-l-d   0 Apr  1 20:21 activation_ta
+nh.type
-rw-r--r--  1 paul matthews mkgroup-l-d 287 Apr  1 20:21 add_edge.data
-rw-r--r--  1 paul matthews mkgroup-l-d  32 Apr  1 20:21 add_edge.type
-rw-r--r--  1 paul matthews mkgroup-l-d   0 Apr  1 20:21 add_graph.typ
+e
-rw-r--r--  1 paul matthews mkgroup-l-d  83 Apr  1 20:21 add_input.dat
+a
-rw-r--r--  1 paul matthews mkgroup-l-d  18 Apr  1 20:21 add_input.typ
+e
-rw-r--r--  1 paul matthews mkgroup-l-d  92 Apr  1 20:21 add_neuron.da
+ta
-rw-r--r--  1 paul matthews mkgroup-l-d  32 Apr  1 20:21 add_neuron.ty
+pe

    .....

    ...l.

    .....
[download]

There are a number of different tests we can add, at the moment the file just has the variable type information, but you can also store the realtime data too. From this you can take random sample and run a select few through the subroutines created as tests so that when a user runs make test they have a more robust suite.

I'm still writing the tests, but this is the second script generalize_tests.plx thus far.
(Update:First version that produces actual tests now up.)

#!/usr/bin/perl

use strict;
use warnings;

#Check command line arguments
die "Incorrect number of arguments\n\tFormat : generalize_tests <file>
+\n\n" if ( $#ARGV != 0 );

my $package = $ARGV[0];
my @bits = split "/", $package;
my $datadir = "";
my $new_package = "";
for ( 0 .. scalar @bits - 2 ) {
    $datadir .= $bits[$_] . "/";
    $new_package .= $bits[$_] . "::";
} 
$new_package .= $bits[scalar @bits - 1];
$new_package =~ s/\.pm//;
print "NEW PACKAGE : $new_package\n";
chop $datadir;

#make a directory to store the tests
my $testdir = "t";
if ( ! -d "$datadir/$testdir" ) { system ("mkdir $datadir/$testdir"); 
+}

#get the list of subroutines
my @files = glob("$datadir/data/*.type");

print "Creating tests for the following Subroutines\n\n";
my $f;
foreach $f ( sort @files ) {
    if ( -s $f ) {
        $f =~ /$datadir\/data\/(\w+).type/;
        my $subroutine = $1;
        print "$subroutine ";
        open SUB, "<$f" or die "Could not open file for read2 : $!\n";
        my @lines = <SUB>;
        close (SUB);
        
        my %hash;
        #Now create a hash of hashes of what each subroutine can take 
+as inputs
        foreach my $l (@lines) {
            chomp $l;
            my @arr = split ',', $l;
            for ( 0 .. scalar @arr - 1) {
                ${$hash{$_}}{$arr[$_]} = 1; 
            }

        }
        foreach my $h ( keys %hash ) {
            #print "[$h] : ";
            foreach my $k ( keys %{$hash{$h}} ) {
            #    print "$k ";
            }
            #print "\n";
        }

        #Now create tests based on the input parameters.
        $f =~ s/\/data\//\/t\//;
        $f =~ s/\.type/\.t/;
        open "TEST", ">$f" or die "Could not open file for write : $!\
+n";
        print TEST <<__END__;
use Test::More tests=> 1; 
use $new_package;

my \$self = $new_package->new();

ok(\$self->$subroutine(), "Testing $subroutine with no args");

__END__


        close (TEST);
    }
}
print "\n\n";
[download]

which gives the following output...

Creating tests for the following Subroutines

GetNewVertexNumber add_edge add_input add_neuron add_output new run_ne
+twork run_node set_edge_weights set_input_value
[download]

Which as you can see makes .t files for the excercised functions.

total 10
drwxr-xr-x+ 2 paul matthews mkgroup-l-d  0 Apr  1 16:32 .
drwxr-xr-x+ 4 paul matthews mkgroup-l-d  0 Apr  1 16:34 ..
-rw-r--r--  1 paul matthews mkgroup-l-d 37 Apr  1 16:34 GetNewVertexNu
+mber.t
-rw-r--r--  1 paul matthews mkgroup-l-d 37 Apr  1 16:34 add_edge.t
-rw-r--r--  1 paul matthews mkgroup-l-d 37 Apr  1 16:34 add_input.t
-rw-r--r--  1 paul matthews mkgroup-l-d 37 Apr  1 16:34 add_neuron.t
-rw-r--r--  1 paul matthews mkgroup-l-d 37 Apr  1 16:34 add_output.t
-rw-r--r--  1 paul matthews mkgroup-l-d 37 Apr  1 16:34 new.t
-rw-r--r--  1 paul matthews mkgroup-l-d 37 Apr  1 16:34 run_network.t
-rw-r--r--  1 paul matthews mkgroup-l-d 37 Apr  1 16:34 run_node.t
-rw-r--r--  1 paul matthews mkgroup-l-d 37 Apr  1 16:34 set_edge_weigh
+ts.t
-rw-r--r--  1 paul matthews mkgroup-l-d 37 Apr  1 16:34 set_input_valu
+e.t
[download]

I'm still writing the test generation code, but wanted to get feedback on the concept and the approach? Are there already 15 modules that do this 10 times better (I'm still going to do it for the mental exercise anyways). Does anyone think that this would be useful or can offer some ways to improve (mainly the concept not my badly written code.

(Update:The main point of this is to see if I can get it to stress my code and point out areas that should or should not fail. Has this worked? YES! On my first set of tests that just runs each subroutine with no Arguments several ran through without aborting as I hadn't added boundary condition code.)

This might not run every module style at this stage, but its given me a better understanding of code structure and argument types along the way.

Regards Paul

P.S.Excuse the long post, but I hope someone will make it to the bottom and find it an interesting read. I'll update the code as I progress.

Comment on Automated Edge Testing Select or Download Code

Replies are listed 'Best First'.
Re: Automated Edge Testing by BrowserUk (Patriarch) on Apr 01, 2005 at 23:56 UTC
You may like to take a look tmoertel's brilliant Test::LecroTest. It looks like you're approaching testing from a similar angle and you may be able to cross fertilise. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. Lingua non convalesco, consenesco et abolesco. Rule 1 has a caveat! -- Who broke the cabal?	[reply]
Re^2: Automated Edge Testing by thekestrel (Friar) on Apr 02, 2005 at 01:10 UTC
BrowserUK, I dropped tom a quick note and am looking into his module. Thanks for the link =). (Update:The main difference with my approach and LectroTest is that it uses property definitions that you define manually and I am trying to infer these automatically from run time data. This still means that you are required to so some manual work, but you can just say I expect this result of that type and it can hound it with a series of tests to make sure this is the case, which would be far more cumbersome to create by hand. My approach can only infer what these characteristics are based on the input data. The more data (quantity and diverisity) you provide the better chance it would have of say inferring that inferring that the return value is generally 1 .. 100 rather than 1 .. 3 because you only supplied it a few samples. Ultimately Tom's module is a more robust method, but requires a little configiring on your the users part to get the best out of it (but, doesn't everything), but as BrowserUK pointed out, similar concepts, slightly different approaches. ) Regards Paul.	[reply]
Re: Automated Edge Testing by pernod (Chaplain) on Apr 04, 2005 at 08:23 UTC
This looks very interesting. Writing tests that are valid for different subroutine signatures will be a challenge, though. Your example above creates a test where each subroutine is excercized with an empy argument list. Generating the input with non-empty argument lists is non-trivial. What happens when you have five subroutines that expect hashrefs and one that expects a scalar? I've pondered this problem a bit earlier, and perhaps it is possible to add more code to parse subroutine signatures and feeding the results to some sort of test factory. This will at least reduce overhead in writing new test types. Your title includes "Edge Testing". By concentrating on edge cases, i assume that you attempt to improve code coverage. Perhaps you could piggyback on Devel::Cover to get statistics on your code? Devel::Cover presents its data in a structured way, which is (normally) used to generate the pretty HTML-presentations. Maybe it is possible to analyse test input, and what branches are excercised, and then to generate more input data to excercise the uncovered branches? This does not sound trivial, but then again, your solution isn't either ;) Lastly, I think your code parsing is a bit fragile. Regular expressions are wonderful, but Perl syntax is notoriously flexible. Therefore, reuse of your work depends on a code standard. This is fine, of course, as powerful tools will put restrictions on what you are allowed to do (viz static analysis tools). Still, Perl hackers are notoriously TIMTOWTDI. Maybe PPI can help with the parsing? Your code is very interesting, and I wish you good luck in your project. Please keep us posted as you progress :) pernod -- Mischief. Mayhem. Soap.	[reply]