Re: Writing tests when you don't know what the output should be

(Example in italics)

How do I write a test to ensure that the module is loading the correct files if I don't yet know what the files in the directory will be?

You create a known set of files in the directory.

Let's assume a simple "pocket calculator module". Create - say three - files "one-plus-two.calc", "two-plus-three.calc", "three-times-five.calc", containing "1+2", "2+3", "3*5".

How do I test whether the module is performing these operations correctly?

You know the input (because you created it), you know the expected output. Compare the output of your module with the expected output. Test::More and friends are extremely useful.

Your test may either have a hardcoded result for each input file, perhaps in a hash (%expected=( "one-plus-two.calc" => 3, "two-plus-three.calc" => 5, ...)), or a result file for each input file ("one-plus-two.result" contains "3", and so on).

Do I need to create a bunch of dummy data structures that mimic all the different possibilities of what the real data will look like and run the tests with this dummy data?

If you want a 100% test, yes. Hint: You can generate the input and the expected output.

To test all basic operations on numbers from one to ten, use three nested loops:

my $testNo=1;
for my $x (1..10) {
  for my $y (1..10) {
    for my $op (qw( + - * / )) {
      write_file("test$testno.calc","$x $op $y");
      write_file("test$testno.result",eval "$x $op $y"); # eval just f
+or demo purposes ...
      $testNo++;
    }
  }
}
[download]

This seems like an awful lot of work and I'm not sure it would really be worth the time.

It depends on how the module is used, and how reliable it has to be. Start with a few simple tests, then extend.

If you find bugs later, FIRST add a test for the bug, THEN fix the bug, and re-run the tests to confirm that the bug is gone. This way, you will never re-introduce the bug, because the test for this bug will fail.

Another hint: Don't test the entire module at once, test its components (functions / methods). So if you have a file searching function (something like glob), write a test that runs that function on a set of files, and compare its return value with the expected result. ((You would expect it to return ( "one-plus-two.calc", "two-plus-three.calc", "three-times-five.calc" ) in the example.) You will likely have a function that parses a file. To test it, call it with a file with known content, and compare the return value with the expected result. (See above: %expected=( "one-plus-two.calc" => { op => '+', left => 1, right => 2 }, ... )). You will have a function that actually works on the parsed data. To test it, call it with known parser output. (use Test::More tests => 3; use Calculator; is(calculate({ op => '+', left => 1, right => 2 }),3,'add one plus two'); ...). You have a function that outputs data. To test it, call it with known input, as usual. (... is(numberToWords(12),"twelve","number to words for 12"); ...)

If your code does everything in one big function: Big chance to clean up your code! ;-) Like in the old unix philosopy, a function should do one thing, and it should do that right. As a rule of thumb, most functions should have drastically less than 100 lines of code.

Still confused? CPAN is full of tests. Look around how modules are tested. DBI and DBD::ODBC have a lots of tests, for every little detail. CGI has a few tests. HTML::Parser has some. Test::Simple has lots of tests, so does Text::CSV_XS. Most other modules come with at least a few tests. Pick a module you know well, and look at its tests.

Alexander

--
Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)

Comment on Re: Writing tests when you don't know what the output should be Select or Download Code

Replies are listed 'Best First'.
Re^2: Writing tests when you don't know what the output should be by nysus (Parson) on May 17, 2016 at 21:01 UTC
Regarding the files, I don't know what the files will be named ahead of time so how do I test if the module is loading the correct ones? If I create dummy files to run tests on, it will interfere with the proper operation of the actual module. $PM = "Perl Monk's"; $MCF = "Most Clueless ~~Friar~~ ~~Abbot~~ ~~Bishop~~ ~~Pontiff~~ ~~Deacon~~ Curate"; $nysus = $PM . ' ' . $MCF; Click here if you love Perl Monks	[reply]
Re^3: Writing tests when you don't know what the output should be by stevieb (Canon) on May 17, 2016 at 21:09 UTC
How do you plan on deciding which files to load then? Will they be defined by name in some sort of configuration? You should be setting up an actual distribution structure. See Module::Starter. All of your tests go in `t/`, and you can then create a test directory structure and associated data in for eg: `t/data/`. Your module should have a method or something that allows a script to tell it what directory to look in for the files (the tests in the `t/*.t` test files would do this). Setting up a proper testing infrastructure allows you to very precisely control everything, and easily and quickly add new sample data and new tests going forward while making changes, or particularly when a bug has been found. Any chance you could show us the code you have so far?	[reply] [d/l] [select]
Re^4: Writing tests when you don't know what the output should be by nysus (Parson) on May 17, 2016 at 21:58 UTC
Sure thing: `package DupEventChecker; use Modern::Perl; use IO::Dir; use Moose; has 'dir' => ( is => 'ro', isa => 'IO::Dir', writer => '_set_dir' ); sub BUILD { my $self = shift; my $args = shift; $self->_set_dir(IO::Dir->new($args->{dir_name})); } return 1;` [download] And here's the script: `use Modern::Perl; use DupEventChecker; my $dir = DupEventChecker->new({dir_name => '.'});` [download] $PM = "Perl Monk's"; $MCF = "Most Clueless ~~Friar~~ ~~Abbot~~ ~~Bishop~~ ~~Pontiff~~ ~~Deacon~~ Curate"; $nysus = $PM . ' ' . $MCF; Click here if you love Perl Monks	[reply] [d/l] [select]
Re^5: Writing tests when you don't know what the output should be by choroba (Cardinal) on May 17, 2016 at 22:24 UTC
Re^6: Writing tests when you don't know what the output should be by nysus (Parson) on May 18, 2016 at 03:54 UTC
Some notes below your chosen depth have not been shown here
Re^6: Writing tests when you don't know what the output should be by nysus (Parson) on May 19, 2016 at 15:43 UTC
Some notes below your chosen depth have not been shown here
Re^5: Writing tests when you don't know what the output should be by stevieb (Canon) on May 17, 2016 at 22:46 UTC
Re^3: Writing tests when you don't know what the output should be by afoken (Chancellor) on May 17, 2016 at 21:08 UTC
Regarding the files, I don't know what the files will be named ahead of time so how do I test if the module is loading the correct ones? If I create dummy files to run tests on, it will interfere with the proper operation of the actual module. Then your module lacks a parameter telling it WHERE to look for the files. During the tests, you want to look for test input files usually below the "t" directory, not where the production files are. Alexander -- Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)	[reply]


Pathologically Eclectic Rubbish Lister
	PerlMonks