Re: test files - organising
by Old_Gray_Bear (Bishop) on Feb 11, 2008 at 20:51 UTC
|
First things first: prove -r or prove -- recurse will chug through the directories looking for *.t files to run.
Second things second: what do you feel comfortable with? Do things you are comfortable with. There is, after all, More Than One Way....
I tend to use the format "tnn-short-memory-jogging-phrase.t", to name my test-files. (ie: t00-basic-api.t, t05-extended-api.t, etc) Each file consists of all of the tests for a single module. I select 'nn' to have common functions grouped together in the prove() output or a directory listing. This works for me until the test-file for a single module gets 'too large', either logically (more that three or four logical/functional groupings) or physically (20+ one-line tests). (I basically follow the same principal that I use in deciding whether some code needs to be broken up into sub-routines -- Do I have to scroll up and down more than half-a-screen? Split it up!) When a test file gets too large, then I build the sub-directory for the module, move the test-harness into it, and refactor into several (much smaller) tnn-aaa.t files.
This has worked reasonably well for me over the years, but it's really a matter of personal taste. Just set things up so that it is easy for you to keep track of things during the Development process.
Also, remember that the Maintenance Programmer has to be able to pick up your test-file philosophy with out too much difficulty later. Make it easy for Them, and They are much more likely to Bless Your Name when they have to fix bugs and make functional changes.
----
I Go Back to Sleep, Now.
OGB
| [reply] [d/l] [select] |
|
|
I have to say that "whatever you feel comfortable with" isn't
really an adequate answer to this kind of question: yeah there's
lots of ways that things can be done, but some are more common
that others, and other things being equal, it's going to be
better to go with "standard practice" rather than making up a new
way of doing things.
That said, Old_Gray_Bear's version of "standard practice" doesn't
sound bad, and might not be a lot different from what I do,
which is:
-
Scatter "t" subdirectories throughout the code tree, with tests
located near the code that they test. So the tests for
Modular/Stuff.pm are in Modular/t.
-
When first writing tests for Modular::Stuff, I name the test file
Modular-Stuff.t. As the tests get more elaborate, I move the
tests into numbered files (e.g. 00-init-Modular-Stuff.t,
01-db_access-Modular-Stuff.t, 02-data_munging-Modular-Stuff.t...).
-
I put data files that tests use in a sub-directory of t called "dat",
and I sometimes put shared code for these tests in modules
located in a sub-directory called "lib".
And yes, prove -r is indeed the way to run it recursively.
It really is a good ideas to get familar with prove, starting
with the man page.
One of the nice things about "prove" by the way, is that you can
narrow down the tests that you run with it by doing things like:
prove '0*.t' '1*.t'
Lately I've been playing around with reserving blocks of numbers
for tests-under-development that aren't expected to work yet.
Tests named '0*.t' or '1*.t' had better work, or something has
come unstuck, but a failure in tests named '2*.t' of '3*.t' just
means you haven't finished something yet.
| [reply] [d/l] [select] |
Re: test files - organising
by adamk (Chaplain) on Feb 11, 2008 at 22:43 UTC
|
It all depends on the size of the project.
http://svn.ali.as/cpan/trunk/PPI/t/
The PPI test suite is flat (with subdirectories for support and data files and a lib for test classes used by various tests) and yet it is quite a large project.
Then again, it's still only 5k-10k SLOC, and it's still a fairly straight forward structure (it's a plain module).
For truly large applications (using one 200k SLOC web application at work as an example) where the test suites are going to be highly diverse, then it's worth looking at breaking things up a bit.
For this project, the root of /t holds bootstrapping and general tests, with subdirectories to hold various classes of tests.
So the root checks all the modules compile, that they pass certain static analysis checks, that the configuration environment loads ok, and that all the external services identified by the configuration code exist and work.
The root is also where we put all the modules that we want to abort the entire test run on failure.
Then we have things like /t/db which contains all the tests for database sanity, DBI code, tests for the data model classes, etc.
Theres a /t/legacy that holds all the tests we've recovered from previous testing efforts that were later abandoned (this code base is over 10 years old now, and has gone through three generations of coding teams).
The /t/inline directory holds all the tests generated by Test::Inline (which we use to scatter tests for assumptions and what not through the codebase).
A /t/cp holds tests for one everything related to one particularly thorny section of the application.
Then theres /t/www for all the web interface testing, and t/dbmigrate for the tests for the database migration system we invented to deal with incredibly painstaking process of migrating an Oracle database that is too large and complex to be able to recreate from scratch.
And finally a /t/cpan to hold additional regression tests for specific bugs in CPAN modules we depend on. Some of these get submitted back to the original modules and removed later.
We also expect to aquire a /t/selenium directory and a /t/endecca (to test a specific third party search engine).
Within each directory, we still do numeric test names (01_foo.t, 02_bar.t).
Apart from straight forward separation of concerns this structure provides, it also makes life easier because you can use things like prove to test one subset of concerns (prove -r -b /t/db for example) without having to run the entire test suite, which for this application now runs out to 20+ minutes.
The directories also have the side-effect of acting as a basic semantic classification, they add more meaning to the test simply by looking at their file names, without having to read documentation. | [reply] |
Re: test files - organising
by Herkum (Parson) on Feb 11, 2008 at 20:06 UTC
|
Take a look at Test::Class. It will make it Easier for you to manage your tests if they are are modules rather than sticking everything into one huge script or dozens of smaller test scripts.
| [reply] |
Re: test files - organising
by frostman (Beadle) on Feb 12, 2008 at 20:39 UTC
|
I recommend subdirectories (and making prove recurse on them). More
specifically, I recommend numbered subdirectories organized by
functional area, where the numbering behaves correctly in standard
globification.
For example, something like this:
000_basics/
001_cpan-prerequisites.t
002_database-bootstrap.t
...
100_orm/
101_dbix-main-schema.t
102_dbix-main-foo.t
...
200_main-classes/
201_My-Special-Foo.t
202_My-Class-Accessor-Lightspeed.t
...
300_parsing/
301_xml-to-spaghetti.t
302_unmelt-ice-cream.t
...
This gives you a few instant advantages:
- Your tests will be run in a useful order (ideally an order of ascending complexity).
- You can sensibly BAIL_OUT if you need to.
- Running only a group of related tests will be easy.
- If you need to later, you can have arbitrary levels of subdirectories without breaking the organization.
- If done right, it makes it easy to find tests.
- If done right, it makes it easy for new devs to know where to put their tests.
Anyway, setting up your test infrastructure is a Really Important Thing
and I urge you to give it the same level of consideration you do to the rest
of your system architecture. People touching your code later, including your future self, will thank you for it.
Oh yeah, and document the test architecture in case you're run over by an elephant and someone else has to figure it out.
isnt( $THEIR::obvious, $YOUR::obvious, "/obvious" );
| [reply] [d/l] [select] |
|
|
Thanks for all the suggestions, it's always obviously a bit tricky trying to figure out how to arrange things when you've no idea how large a project is going to be.
This one is no doubt gonna get VERY large, re-factoring code that's over 8 years old into sensible modules.
Thanks again!
| [reply] |
Re: test files - organising
by DrHyde (Prior) on Feb 12, 2008 at 11:04 UTC
|
I don't know whether prove will enter sub-directories, but I can easily find out. Did you bother to try it out, or read the man page? If not, why not? | [reply] |
|
|
Of course it will.
prove -r
| [reply] |
|
|
Why not make up an answer?
| [reply] |