Time consuming tests

jplindstrom has asked for the wisdom of the Perl Monks concerning the following question:

For the current project I have a test suite in .../t that prove runs in about 14 seconds. That's usable.

But I just added a test that takes a bit longer. It makes SSH connections, and that's the functionality I want to test so it would be useless to mock it just to make it go quicker.

Now the tests take 29 seconds to run. That's too much.

I have this idea of putting these tests in another directory so I easily can chose which tests to run. It could be .../tt, or .../t/slow.

Thoughts?

Update:

Maybe I should have mentioned: This isn't a CPAN module, it's an application at work. Hence, the SSH stuff is kind of dependent on our dev/test environment, and not many people will be installing it. In that regard, it makes sense to skip those tests by default so they aren't run in the production environment.

Thanks all!

Comment on Time consuming tests Select or Download Code

Replies are listed 'Best First'.
Re: Time consuming tests by grinder (Bishop) on Jan 27, 2005 at 11:15 UTC
I have this idea of putting these tests in another directory so I easily can chose which tests to run erk. I would just go with an environment variable: `SKIP: { skip '$ENV{WANT_HONKING_BIG_TEST} is not set', 1 unless exists $ENV{WANT_HONKING_BIG_TEST}; cmp_ok( massive_test, '==', 42, 'blah blah blah' ); }` [download] You can define the environment variable in your shell startup file, so that you always run the test. Someone who has a passing interest in the module downloads it from CPAN and will not pay the cost of the lengthy tests. If they encounter a bug and are diligent about it, they'll read the "Reporting bugs" section in your documentation, and learn that there are additional tests that they can run which may help you diagnose the problem more accurately. I dislike modules that ask "do you want to run these extra tests?". If you have to ask, then you shouldn't run them. People who are in the know and want those extra tests will know how to set up the environment before running CPAN/CPANP in order to get them. - another intruder with the mooring in the heart of the Perl	[reply] [d/l]
Re^2: Time consuming tests by thor (Priest) on Jan 27, 2005 at 12:59 UTC
People who are in the know and want those extra tests will know how to set up the environment before running CPAN/CPANP in order to get them. I would argue the counterpoint. The tests should run by default, and for those people to whom an additional 15 seconds is too much, they can set the environment variable. The only reason I could see to skip the test is in a development phase when you're running your test suite after every minor change. You'd still want to run the entire suite before commiting the change, though. thor `Feel the white light, the light within Be your own disciple, fan the sparks of will For all of us waiting, your kingdom will come`	[reply]
Re^2: Time consuming tests by eyepopslikeamosquito (Archbishop) on Jan 27, 2005 at 22:20 UTC
`skip '$ENV{WANT_HONKING_BIG_TEST} is not set', 1` [download] As for a standard name (it's arbitrary, so why not agree on a standard name?) for said environment variable, this has been discussed a number of times on perl-qa mailing list here and here and here and in a few other places as well (discussion triggered by Skip tests if/unless $ENV{FOO} is set using Test::More). Alas, I don't think a consensus was ever reached. :-( Maybe if someone suggested your `WANT_HONKING_BIG_TEST` we might have got a consensus. :-)	[reply] [d/l] [select]
Re: Time consuming tests by borisz (Canon) on Jan 27, 2005 at 12:51 UTC
I would use a environ var as grinder suggested. I never move a test out of the test directory. However, consider that a test that is never run is useless and can be removed. And is the test really run so often that you can not afford the additional 30 sec's? Boris	[reply]
Re^2: Time consuming tests by jplindstrom (Monsignor) on Jan 27, 2005 at 13:30 UTC
Yes, these are development tools, not an install-sanity check. They should be run like a trillion times a day. Mostly I run a particular .t file when developing that feature but sometimes during this time, and before check-in, I run the entire suite. I went with the suggested SKIP on variable approach and it works fine (thanks grinder!) Depending on how close to the SSH stuff that last edit was, I'm gonna run those tests as well. /J	[reply]