Interactivity in tests - done correctly?

tirwhan has asked for the wisdom of the Perl Monks concerning the following question:

Hello monkses,

I'm currently finishing off a module I plan to release on CPAN. It would be beneficial if the test suite included tests against a live server (simulated tests are also included, but not as good as the real thing). For this I need to prompt the user for some information, to see whether he has such a server available and what it's address etc. are.

I gather that putting interactivity into the general test suite is a big no-no (it breaks the automated smoke tests for one), so I've been thinking about how I could do this and come up with the following:

use Test::More qw(no_plan);

use strict;
use warnings;

SKIP: {
    skip qq(Won't run automatically, execute "export MyModOPTRUN=yes" 
+to enable interactive test)
        unless ($ENV{MyModOPTRUN} && $ENV{MyModOPTRUN} eq "yes");
    warn "\nEnter your innermost secret:\n";
    my $secret = (<STDIN>);
    ok ($secret,"Yep, that's it");
}
[download]

Obviously replacing "MyMod" with the actual name of my module. This will produce

t/02.optional....ok
        1/1 skipped: Not running automatically, execute "export MyModO
+PTRUN=yes" to enable interactive test
All tests successful, 1 subtest skipped.
Files=1, Tests=1,  0 wallclock secs ( 0.00 cusr +  0.01 csys =  0.01 C
+PU)
[download]

When run without the environment variable set, which should be all right I think.

Are there any problems with this approach? Does the "export" syntax even work on non-*NIX-shells? If not, is there an equivalent (I can test for $^O and adjust the skip message accordingly)? Is there a better way to do this?

Thanks in advance

All dogma is stupid.

Comment on Interactivity in tests - done correctly? Select or Download Code

Replies are listed 'Best First'.
Re: Interactivity in tests - done correctly? by moritz (Cardinal) on Jan 22, 2009 at 13:11 UTC
First of all I think that environment variables are the right tool for the job, although I'd call them `TEST_INTERACTIVE` or something like that. I'd also provide the ability to provide all information via a config file or environment variables, because the value of a test suite is that you can run it with very little effort - which gets lost if you have to enter the same things all over again. Does the "export" syntax even work on non-*NIX-shells? It doesn't even work on some Unix shells, for example on csh. You could just say "set the environment MyModOPTRUN variable to yes", and hope that the user know how to do it.	[reply] [d/l]
Re: Interactivity in tests - done correctly? by Corion (Patriarch) on Jan 22, 2009 at 13:07 UTC
Why ask the user for the login/password, when the user could set `$ENV{MyModOPTRUN}` to it? I would output `'$ENV{MyModOPTRUN} not set, not running interactive tests'` as the status message and have the instructions how to enable it in README or INSTALL. The "export" syntax only works in unixish shells, but somebody who wants to run the interactive tests should know how to do it for the respective OS ;)	[reply] [d/l] [select]
Re^2: Interactivity in tests - done correctly? by tirwhan (Abbot) on Jan 22, 2009 at 13:19 UTC
There is a total of three strings I need from the user, I think this is easier done interactively than having them set three different environments variables. Also, I'd like to give them some information about the nature of the test they will be running. You're right, this can be put into the README or INSTALL files, but nobody reads the damn things ;-). somebody who wants to run the interactive tests should know how to do it for the respective OS ;) True, and moritz makes an excellent point about it not even being the same on csh. I'll probably just tell them to set the environment variable and provide further assistance in the README file which they can read if they need it. Thanks! All dogma is stupid.	[reply] [d/l]
Re^3: Interactivity in tests - done correctly? by leocharre (Priest) on Jan 22, 2009 at 20:09 UTC
I read the README and INSTALL files :-) One of the things that can piss me off to no end is watching cli cpan hang because a) the config on that machine is missing a timeout b) you didn't code one I would say code it, but provide a timeout, and let the thing install even if it's missing something. Maybe at compile time it could break and tell why. If your module becomes a dep for something else, maybe whomever is installing it will already be aware of what's up.	[reply]
Re^4: Interactivity in tests - done correctly? by tirwhan (Abbot) on Jan 22, 2009 at 20:37 UTC
Re: Interactivity in tests - done correctly? by mje (Curate) on Jan 22, 2009 at 13:39 UTC
On the issue of prompting under test conditions: If you use ExtUtils::MakeMaker to build your module you can import "prompt" http://search.cpan.org/~mschwern/ExtUtils-MakeMaker-6.48/lib/ExtUtils/MakeMaker.pm#Other_Handy_Functions: `use ExtUtils::MakeMaker 5.16, qw(prompt);` [download] and as far as I am aware the prompt returns the default if no stdin or running in an automated system - like smoking.	[reply] [d/l]
Re^2: Interactivity in tests - done correctly? by Anonymous Monk on Jan 22, 2009 at 14:35 UTC
Even if you don't use ExtUtils::MakeMaker, you should use ExtUtils::MakeMaker::prompt, since every perl comes with ExtUtils::MakeMaker.	[reply]
Re^3: Interactivity in tests - done correctly? by mje (Curate) on Jan 22, 2009 at 16:20 UTC
Agreed. I slightly misread the post at first or I would have made that response use prompt.	[reply]
Re^2: Interactivity in tests - done correctly? by Sandy (Curate) on Jan 22, 2009 at 19:40 UTC
I am a little uncertain as what you mean. If I use 'prompt' in the Makefile.PL, how can the test scripts get this information. I am having a similar requirement as the OP, in that I need username and password to test. I do not want to use an environment variable for the password (although it is a development database, the DBA is very strict about keeping even those passwords very secure).	[reply]
Re^3: Interactivity in tests - done correctly? by mje (Curate) on Jan 23, 2009 at 10:37 UTC
Yes, as I said I initially slightly misread the question and hence the reference to Makefile.PL. You can use ExtUtils::MakeMaker->prompt in your test scripts too. `$ perl -MExtUtils::MakeMaker -le 'print ExtUtils::MakeMaker::prompt("f +red", "default");' fred [default] # I just hit CR here default` [download]	[reply] [d/l]
Re^2: Interactivity in tests - done correctly? by tirwhan (Abbot) on Jan 22, 2009 at 20:28 UTC
Thanks! I came across ExtUtils::MakeMaker::prompt when looking for answers to this question prior to posting (should have mentioned that). Correct me if I'm wrong, but I was under the impression that using E::MM::prompt means that a.) Anyone installing the module via Module::Build would not be prompted? b.) Everyone installing the module interactively (via Makefile.PL) would be prompted? I don't want either of these, I like to provide both a Build.PL and Makefile.PL so people have the choice of install system to use. And I also don't want to prompt every user for this information, I just want to tell the user "you can test this against against a live service if you want, and if you tell me to do so I'll prompt you for the information that's required". Not saying your suggestion is a bad one, but if both of these assumptions are true it's just not the right fit for me. All dogma is stupid.	[reply] [d/l]
Re^3: Interactivity in tests - done correctly? by Anonymous Monk on Jan 23, 2009 at 07:31 UTC
ExtUtils::MakeMaker::prompt is the de-facto standard for prompt and Module::Build follows the convention. http://search.cpan.org/dist/Module-Build/lib/Module/Build/API.pod#prompt. If you want it to be optional, you can always `perl ....PL --promptme if( $promptme ){ ...prompt(... } # OR if( $ENV{prompt_me_big_boy} ){ ...prompt(... }` [download]	[reply] [d/l]