http://qs1969.pair.com?node_id=399644

jfroebe has asked for the wisdom of the Perl Monks concerning the following question:

Hi,

I've been tasked on improving (or replacing) a custom test suite of bourne shell, c code, and cshell code.

I've been looking around and the Test::* modules appear that to provide what I'm looking for: a relatively simple framework to test arbitrary conditions that may or may not be related to perl itself. The output needs to be simple to the point of just having to eye ball it.

For example, testing a new version of a DBMS with a set of queries to determine if the particular version will cause problems.

My questions are:

1) has anyone done something similar with the Test::* modules?
2) is there something better to use? (I would prefer to use perl)
3) any issues I should be aware of

thanks

Jason L. Froebe

No one has seen what you have seen, and until that happens, we're all going to think that you're nuts. - Jack O'Neil, Stargate SG-1

Replies are listed 'Best First'.
Re: using Test::* modules for generic testing of non perl stuff?
by dragonchild (Archbishop) on Oct 15, 2004 at 20:30 UTC
    I've used Test::Cmdline to test a C program. Apache::Test is excellent to test pretty much any Apache usage. Test::More + LWP will test anything you can talk to over the internet.

    Is that what you're asking about?

    Being right, does not endow the right to be rude; politeness costs nothing.
    Being unknowing, is not the same as being stupid.
    Expressing a contrary opinion, whether to the individual or the group, is more often a sign of deeper thought than of cantankerous belligerence.
    Do not mistake your goals as the only goals; your opinion as the only opinion; your confidence as correctness. Saying you know better is not the same as explaining you know better.

      Actually that and more...

      For example:

      1) verify all necessary dbms client and server files are in a directory, else fail
      2) verify versions of client and server binaries & libraries
      3) run a series of queries that pass/fail where desired
      4) verify 3rd party binaries work correctly with the dbms server
      5) verify os patch and kernel options
      6) etc.

      This one example would be used to certify a particular dbms version with the local build of the dbms and the local network

      I'm doing about 1/2 this by hand now and since I'm lazy, I don't want to keep doing it. This is just an example of a whole series of other applications I wish to put Test::* towards.

      Jason L. Froebe

      No one has seen what you have seen, and until that happens, we're all going to think that you're nuts. - Jack O'Neil, Stargate SG-1

        You could do pretty much all of this with Perl and Test:: modules if you want. The benefit of Test:: modules is that you get nice output which can be run as a single file, or under something like Test::Harness which gives you a report for a group of tests.

        1) verify all necessary dbms client and server files are in a directory, else fail

        Write a perl script which reads in the directories of these and then check those values against a static list you constructed. This can be done with Test::More::is_deeply if you have them in order, or Test::More::eq_set if they are not in order (remember though that eq_set is not a Test function and must be wrapped with Test::More::ok).

        There may also be a Test:: module out there for specifically checking file existence, I am not sure since I have never had the need. And if there isn't, you could always write one ;-)

        2) verify versions of client and server binaries & libraries

        I would expect that a combination of file test and shell commands (to libtool or something) would get you the answers you want to know, then just test those answers against your expected values using Test::More

        3) run a series of queries that pass/fail where desired

        Load DBI and run some tests, checking the output with Test::More

        4) verify 3rd party binaries work correctly with the dbms server

        I am not sure what this entails, but again, I am sure some shell commands using backticks or system() would get you the answers.

        6) verify os patch and kernel options

        see the last three answers above ...

        Really you can do just about anything with perl assuming you have some kind of programattic access to it. And with a little work, you can hook it into Test::More.

        And as I said above, if you develop any new and possibly useful Test:: modules, put them on CPAN and share the wealth.

        -stvn
      Better yet, use Test-WWW-Mechanize instead of LWP::UserAgent. It's a proper subclass of WWW::Mechanize, which is a proper subclass of LWP::UserAgent, but has some fun testing stuff. It's nice to be able to call, for example
      my $agent = Test::WWW::Mechanize( autocheck => 1 ); $agent->get( $homepage ); $agent->html_ok( "Valid HTML" );

      xoxo,
      Andy

Re: using Test::* modules for generic testing of non perl stuff?
by biosysadmin (Deacon) on Oct 15, 2004 at 20:35 UTC
    If your stuff is command-line based, why not just use backticks to capture the output of some shell commands in your test scripts? Simply generate some acceptable output and compare it to the dynamic output generated by your test script. I've misused the Test::* modules to test non-code things (such as DNS MX configurations) and it's usually very effective.

    Can you be more specific on exactly what you're trying to do, how you want to do things and what you've tried? I think this is a great idea, it's just difficult to give advice without more specifics.

      some is command line based but I'm trying to avoid calling os commands directly if possible (hetergenous environment where output isn't always the same for the same commands). I would prefer to use pure perl where possible.

      Can you elaborate on your misuse of the Test::* modules for testing non code things? Specifically why it was 'misused'. I'm trying to come up with pros and cons as well for the use of Test::* for non-code things.

      Jason L. Froebe

      No one has seen what you have seen, and until that happens, we're all going to think that you're nuts. - Jack O'Neil, Stargate SG-1

        I used the word "misuse" because I think that the Test::* modules were originally used for testing code, and I've adapted them to some orthogonal uses.

        At it's most fundamental, you just need to define a set of "things" you want to test and define their acceptable output. This can be tricky, as you might need to define a range of acceptable output values. One strategy that I've used in the past is to predefine a hash like this:

        %commands = { $cmd1 => $output1, ... };
        Then execute the commands and test the dynamic output against the predefined output.

        I've actually got to run and catch a plane, but I'll be sure to check on this thread when I get back. Cheers. :)

Re: using Test::* modules for generic testing of non perl stuff?
by data64 (Chaplain) on Oct 16, 2004 at 02:28 UTC

    For example, testing a new version of a DBMS with a set of queries to determine if the particular version will cause problems.

    At work, some folks use SqlUnit to do just that. It does seem to be a better tool to do database kind of things than any perl based testing tool I have seen.


    Just a tongue-tied, twisted, earth-bound misfit. -- Pink Floyd

      It looks good but is Java based... Java is rarely installed on any of our server or client (unix/linux) machines. Unfortunately, that rules out the use of java. :-(

      No one has seen what you have seen, and until that happens, we're all going to think that you're nuts. - Jack O'Neil, Stargate SG-1

Re: using Test::* modules for generic testing of non perl stuff?
by tstock (Curate) on Oct 16, 2004 at 15:47 UTC
    I use the perl Test framework to test lots-a-things. These are all crude test examples below -
    # check webservice like( `lynx http://tiago.com -head -dump`, qr/200 OK/, 'HEAD request using lynx' ); # disk space for (`df`) { my ($filesystem, $use) = /^(\S+) .+ (\d+)%/; next unless ($filesystem && $use); ok($use < 70, "$use \%filesystem"); } # running daemons my @ps = `ps -e`; for my $daemon (qw(crond syslogd ntpd)) { my $cnt = grep /$daemon/, @ps; ok($cnt, "$daemon ($cnt)"); } # software version like( `nmap --version`, qr/\W3\.\d/, 'nmap version test' ); # binaries and path for my $bin ('ping','kill','pod2text', 'nslookup', 'tail') { my $loc = `which $bin`; chomp $loc; ok(stat($loc), $loc); ok(-x _, 'executable by effective uid/gid'); } # ping the machine $ip = '66.39.54.27'; my $ret = join('', `ping -c 5 -W 1 $ip`); like($ret, qr/ 0\% packet loss/, 'packet loss'); + # DNS $ip = '66.39.54.27'; $name = 'perlmonks.org'; like(`host $ip`, qr/$name/, 'DNS'); like(`host $name`, qr/$ip/, 'reverse DNS');
    It's also not hard to make other languages output in the format usable by Test::Harness, so incidently, if anyone care to make a shell library of functions that mimic the Test::More module, drop me a note, I'm interested.

    Tiago
      Something I whipped up just this morning. It should work with Test::Harness, it may be missing some of the Test::More methods, let me know if I've mised the mark or something else needs to be added.
      #!/bin/bash # This is a shell library for interfacing with Perl's Test::Harness fr +amework # Copyright 2004. Michael Grubb SHOULD_PRINT_PLAN=0 TEST_PLAN=0 TEST_COUNTER=0 MATCH_PROG=/bin/egrep MATCH_ARGS="-q" plan() { if [ "${1}" == "no_plan" ] ; then SHOULD_PRINT_PLAN=1 TEST_PLAN=0 else TEST_PLAN="${1}" SHOULD_PRINT_PLAN=1 _printPlan SHOULD_PRINT_PLAN=0 fi } endtests() { _printPlan } ok() { _updatePlan if [ $1 != 0 ]; then echo -n "not " fi echo "ok ${TEST_COUNTER} ${2} # \$1 is '$1'" } is() { _updatePlan if [ $1 != ${2} ] ; then echo -n "not " fi echo "ok ${TEST_COUNTER} ${3} # expecting '$1' got '$2'" } match() { _updatePlan echo "$2" | ${MATCH_PROG} ${MATCH_ARGS} ${1} rv=$? if [ $rv -ne 0 ] ; then echo -n "not " fi echo -n "ok ${TEST_COUNTER} ${3} " if [ $rv -ne 0 ] ; then echo "# '$2' did not match '$1'" else echo fi } diag() { echo "# $1" 1>&2 } _updatePlan() { TEST_COUNTER=$((${TEST_COUNTER}+1)) if [ ${SHOULD_PRINT_PLAN} ] ; then TEST_PLAN=${TEST_COUNTER} ; fi } _printPlan() { if [ ${SHOULD_PRINT_PLAN} -eq 1 ] ; then echo "1..${TEST_PLAN}" fi }

      And here is a script that shows it's usage:

      #!/bin/bash . ./Test.sh plan "no_plan" diag "# TEST_PLAN=$TEST_PLAN" ok 0 "Test 1" ok 0 "Test 2" ok 2 "Test 3" is "this" "this" is "this" "that" is 1 0 is 0 1 is 1 1 match "^ab[cd]efg$" "abcefg" match "^ab[cd]efg$" "abdefg" match "^ab[cd]efg$" "abcdefg" match "^ab[cd]efg$" "abhefg" endtests # only needed when "no_plan" is used
      Hope this helps. Oh, and by default the "match" function uses egrep -q but you can override that if you set MATCH_PROG and MATCH_ARGS after you source Test.sh. Let me know if this is useful. Also note that getting Test::Harness to run this is an excercise left to the reader.
Re: using Test::* modules for generic testing of non perl stuff?
by jplindstrom (Monsignor) on Oct 16, 2004 at 14:10 UTC
    At work I have used Test::More and family to build a framework for unit testing an application server + business logic modules.

    It involved making xml calls to the server and see that the correct values are returned, and that the database contents changes accordlingly.

    For this I wrote a generic Perl client to call the application server, Test::More for the returned values and Test::DatabaseRow for testing the database. The rest looks like any ordinary .t script.

    This iterative, low ceremony setup proved to be orders of magnitude more productive than writing unit tests in C++.

    /J

Re: using Test::* modules for generic testing of non perl stuff?
by petdance (Parson) on Nov 30, 2004 at 05:26 UTC
    The things you're talking about are the very reason that I created prove, included with Test::Harness. Take a look at some of my testing talks at http://petdance.com/perl/. Also see http://qa.perl.org.

    Currently I'm testing Perl and PHP with a whole ton of *.t files.

    xoxo,
    Andy

Re: using Test::* modules for generic testing of non perl stuff?
by petdance (Parson) on Nov 30, 2004 at 05:34 UTC
    Plus, have you been to a Chicago Perl Mongers (http://chicago.pm.org) meeting? We're a good helpful bunch, led by a testing zealot.:-)

    xoxo,
    Andy