jfroebe has asked for the wisdom of the Perl Monks concerning the following question:
Hi,
I've been tasked on improving (or replacing) a custom test suite of bourne shell, c code, and cshell code.
I've been looking around and the Test::* modules appear that to provide what I'm looking for: a relatively simple framework to test arbitrary conditions that may or may not be related to perl itself. The output needs to be simple to the point of just having to eye ball it.
For example, testing a new version of a DBMS with a set of queries to determine if the particular version will cause problems.
My questions are:
1) has anyone done something similar with the Test::* modules?
2) is there something better to use? (I would prefer to use perl)
3) any issues I should be aware of
thanks
Jason L. Froebe
No one has seen what you have seen, and until that happens, we're all going to think that you're nuts. - Jack O'Neil, Stargate SG-1
Re: using Test::* modules for generic testing of non perl stuff?
by dragonchild (Archbishop) on Oct 15, 2004 at 20:30 UTC
|
| [reply] |
|
Actually that and more...
For example:
1) verify all necessary dbms client and server files are in a directory, else fail
2) verify versions of client and server binaries & libraries
3) run a series of queries that pass/fail where desired
4) verify 3rd party binaries work correctly with the dbms server
5) verify os patch and kernel options
6) etc.
This one example would be used to certify a particular dbms version with the local build of the dbms and the local network
I'm doing about 1/2 this by hand now and since I'm lazy, I don't want to keep doing it. This is just an example of a whole series of other applications I wish to put Test::* towards.
Jason L. Froebe
No one has seen what you have seen, and until that happens, we're all going to think that you're nuts. - Jack O'Neil, Stargate SG-1
| [reply] |
|
You could do pretty much all of this with Perl and Test:: modules if you want. The benefit of Test:: modules is that you get nice output which can be run as a single file, or under something like Test::Harness which gives you a report for a group of tests.
1) verify all necessary dbms client and server files are in a directory, else fail
Write a perl script which reads in the directories of these and then check those values against a static list you constructed. This can be done with Test::More::is_deeply if you have them in order, or Test::More::eq_set if they are not in order (remember though that eq_set is not a Test function and must be wrapped with Test::More::ok).
There may also be a Test:: module out there for specifically checking file existence, I am not sure since I have never had the need. And if there isn't, you could always write one ;-)
2) verify versions of client and server binaries & libraries
I would expect that a combination of file test and shell commands (to libtool or something) would get you the answers you want to know, then just test those answers against your expected values using Test::More
3) run a series of queries that pass/fail where desired
Load DBI and run some tests, checking the output with Test::More
4) verify 3rd party binaries work correctly with the dbms server
I am not sure what this entails, but again, I am sure some shell commands using backticks or system() would get you the answers.
6) verify os patch and kernel options
see the last three answers above ...
Really you can do just about anything with perl assuming you have some kind of programattic access to it. And with a little work, you can hook it into Test::More.
And as I said above, if you develop any new and possibly useful Test:: modules, put them on CPAN and share the wealth.
| [reply] |
|
|
|
|
Better yet, use Test-WWW-Mechanize instead of LWP::UserAgent. It's a proper subclass of WWW::Mechanize, which is a proper subclass of LWP::UserAgent, but has some fun testing stuff. It's nice to be able to call, for example
my $agent = Test::WWW::Mechanize( autocheck => 1 );
$agent->get( $homepage );
$agent->html_ok( "Valid HTML" );
| [reply] [d/l] |
Re: using Test::* modules for generic testing of non perl stuff?
by biosysadmin (Deacon) on Oct 15, 2004 at 20:35 UTC
|
If your stuff is command-line based, why not just use backticks to capture the output of some shell commands in your test scripts? Simply generate some acceptable output and compare it to the dynamic output generated by your test script. I've misused the Test::* modules to test non-code things (such as DNS MX configurations) and it's usually very effective.
Can you be more specific on exactly what you're trying to do, how you want to do things and what you've tried? I think this is a great idea, it's just difficult to give advice without more specifics. | [reply] |
|
some is command line based but I'm trying to avoid calling os commands directly if possible (hetergenous environment where output isn't always the same for the same commands). I would prefer to use pure perl where possible.
Can you elaborate on your misuse of the Test::* modules for testing non code things? Specifically why it was 'misused'. I'm trying to come up with pros and cons as well for the use of Test::* for non-code things.
Jason L. Froebe
No one has seen what you have seen, and until that happens, we're all going to think that you're nuts. - Jack O'Neil, Stargate SG-1
| [reply] |
|
I used the word "misuse" because I think that the Test::* modules were originally used for testing code, and I've adapted them to some orthogonal uses.
At it's most fundamental, you just need to define a set of "things" you want to test and define their acceptable output. This can be tricky, as you might need to define a range of acceptable output values. One strategy that I've used in the past is to predefine a hash like this:
%commands = {
$cmd1 => $output1, ...
};
Then execute the commands and test the dynamic output against the predefined output.
I've actually got to run and catch a plane, but I'll be sure to check on this thread when I get back. Cheers. :) | [reply] [d/l] |
|
Re: using Test::* modules for generic testing of non perl stuff?
by data64 (Chaplain) on Oct 16, 2004 at 02:28 UTC
|
For example, testing a new version of a DBMS with a set of queries to determine if the particular version will cause problems.
At work, some folks use SqlUnit to do just that. It does seem to be a better tool to do database kind of things than any perl based testing tool I have seen.
Just a tongue-tied, twisted, earth-bound misfit. -- Pink Floyd
| [reply] |
|
It looks good but is Java based... Java is rarely installed on any of our server or client (unix/linux) machines. Unfortunately, that rules out the use of java. :-(
No one has seen what you have seen, and until that happens, we're all going to think that you're nuts. - Jack O'Neil, Stargate SG-1
| [reply] |
Re: using Test::* modules for generic testing of non perl stuff?
by tstock (Curate) on Oct 16, 2004 at 15:47 UTC
|
I use the perl Test framework to test lots-a-things. These are all crude test examples below -
# check webservice
like(
`lynx http://tiago.com -head -dump`,
qr/200 OK/,
'HEAD request using lynx'
);
# disk space
for (`df`)
{
my ($filesystem, $use) = /^(\S+) .+ (\d+)%/;
next unless ($filesystem && $use);
ok($use < 70, "$use \%filesystem");
}
# running daemons
my @ps = `ps -e`;
for my $daemon (qw(crond syslogd ntpd))
{
my $cnt = grep /$daemon/, @ps;
ok($cnt, "$daemon ($cnt)");
}
# software version
like(
`nmap --version`,
qr/\W3\.\d/,
'nmap version test'
);
# binaries and path
for my $bin ('ping','kill','pod2text', 'nslookup', 'tail')
{
my $loc = `which $bin`;
chomp $loc;
ok(stat($loc), $loc);
ok(-x _, 'executable by effective uid/gid');
}
# ping the machine
$ip = '66.39.54.27';
my $ret = join('', `ping -c 5 -W 1 $ip`);
like($ret, qr/ 0\% packet loss/, 'packet loss');
+
# DNS
$ip = '66.39.54.27';
$name = 'perlmonks.org';
like(`host $ip`, qr/$name/, 'DNS');
like(`host $name`, qr/$ip/, 'reverse DNS');
It's also not hard to make other languages output in the format usable by Test::Harness, so incidently, if anyone care to make a shell library of functions that mimic the Test::More module, drop me a note, I'm interested.
Tiago
| [reply] [d/l] |
|
Something I whipped up just this morning. It should work with Test::Harness, it may be missing some of the Test::More methods, let me know if I've mised the mark or something else needs to be added.
#!/bin/bash
# This is a shell library for interfacing with Perl's Test::Harness fr
+amework
# Copyright 2004. Michael Grubb
SHOULD_PRINT_PLAN=0
TEST_PLAN=0
TEST_COUNTER=0
MATCH_PROG=/bin/egrep
MATCH_ARGS="-q"
plan()
{
if [ "${1}" == "no_plan" ] ; then
SHOULD_PRINT_PLAN=1
TEST_PLAN=0
else
TEST_PLAN="${1}"
SHOULD_PRINT_PLAN=1 _printPlan
SHOULD_PRINT_PLAN=0
fi
}
endtests()
{
_printPlan
}
ok()
{
_updatePlan
if [ $1 != 0 ]; then
echo -n "not "
fi
echo "ok ${TEST_COUNTER} ${2} # \$1 is '$1'"
}
is()
{
_updatePlan
if [ $1 != ${2} ] ; then
echo -n "not "
fi
echo "ok ${TEST_COUNTER} ${3} # expecting '$1' got '$2'"
}
match()
{
_updatePlan
echo "$2" | ${MATCH_PROG} ${MATCH_ARGS} ${1}
rv=$?
if [ $rv -ne 0 ] ; then
echo -n "not "
fi
echo -n "ok ${TEST_COUNTER} ${3} "
if [ $rv -ne 0 ] ; then
echo "# '$2' did not match '$1'"
else
echo
fi
}
diag()
{
echo "# $1" 1>&2
}
_updatePlan()
{
TEST_COUNTER=$((${TEST_COUNTER}+1))
if [ ${SHOULD_PRINT_PLAN} ] ; then TEST_PLAN=${TEST_COUNTER} ; fi
}
_printPlan()
{
if [ ${SHOULD_PRINT_PLAN} -eq 1 ] ; then
echo "1..${TEST_PLAN}"
fi
}
And here is a script that shows it's usage:
#!/bin/bash
. ./Test.sh
plan "no_plan"
diag "# TEST_PLAN=$TEST_PLAN"
ok 0 "Test 1"
ok 0 "Test 2"
ok 2 "Test 3"
is "this" "this"
is "this" "that"
is 1 0
is 0 1
is 1 1
match "^ab[cd]efg$" "abcefg"
match "^ab[cd]efg$" "abdefg"
match "^ab[cd]efg$" "abcdefg"
match "^ab[cd]efg$" "abhefg"
endtests # only needed when "no_plan" is used
Hope this helps.
Oh, and by default the "match" function uses egrep -q but you can override that if you set MATCH_PROG and MATCH_ARGS after you source Test.sh. Let me know if this is useful.
Also note that getting Test::Harness to run this is an excercise left to the reader. | [reply] [d/l] [select] |
Re: using Test::* modules for generic testing of non perl stuff?
by jplindstrom (Monsignor) on Oct 16, 2004 at 14:10 UTC
|
At work I have used Test::More and family to build a framework for unit testing an application server + business logic modules.
It involved making xml calls to the server and see that the correct values are returned, and that the database contents changes accordlingly.
For this I wrote a generic Perl client to call the application server, Test::More for the returned values and Test::DatabaseRow for testing the database. The rest looks like any ordinary .t script.
This iterative, low ceremony setup proved to be orders of magnitude more productive than writing unit tests in C++.
/J
| [reply] |
Re: using Test::* modules for generic testing of non perl stuff?
by petdance (Parson) on Nov 30, 2004 at 05:26 UTC
|
| [reply] |
Re: using Test::* modules for generic testing of non perl stuff?
by petdance (Parson) on Nov 30, 2004 at 05:34 UTC
|
Plus, have you been to a Chicago Perl Mongers (http://chicago.pm.org) meeting? We're a good helpful bunch, led by a testing zealot.:-)
| [reply] |
|
|