Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

Timeout on prove

by chrestomanci (Priest)
on Jul 02, 2014 at 13:45 UTC ( [id://1091994]=perlquestion: print w/replies, xml ) Need Help??

chrestomanci has asked for the wisdom of the Perl Monks concerning the following question:

Greetings wise brothers. I seek your wisdom in how to call a halt to testing when nothing useful will be obtained.

I am working on a fairly mature project with a large suite of unit tests, that are run automatically before every merge. Recently there was an interaction between a buggy unit test, and another change to the code base, that caused the unit test to enter an infinite loop and run for many hours before someone killed it. (The offending test normally takes about 10 seconds). It was not noticed sooner because the test run did not actually fail.

Once I found the problem, I naturally wanted to prevent it happening again, and started looking for a timeout option in prove. I was surprised not to find one.

Is there an undocumented or hidden option or idom that I can use with prove to set a timeout, either for each test, or an overall timeout for the whole run?

Replies are listed 'Best First'.
Re: Timeout on prove
by LanX (Saint) on Jul 02, 2014 at 14:05 UTC
    Sorry dunno about options in prove ...

    ... but if you are calling prove automatically you should be able to script a wrapper¹ with a timeout.

    The advantage would be that this should also be a good place to hard-code corporate wide² policies. :)

    Cheers Rolf

    (addicted to the Perl Programming Language)

    footnotes

    ¹) like "merge_prove.pl" :)

    ²) and project specific

Re: Timeout on prove
by andal (Hermit) on Jul 02, 2014 at 14:08 UTC

    Hm. How about simply wrapping call to 'prove' into perl script which uses 'alarm' to stop testing process after certain time?

    Actually, it makes more sense, to put time-outs in each of the tests. But again, who cares about quality of the tests? I was always wondering, who tests the test scripts :)

Re: Timeout on prove
by tobyink (Canon) on Jul 02, 2014 at 19:24 UTC

    Install Time::Limit and then run:

    prove -MTime::Limit=60 -Ilib /path/to/tests

    (This works less well if you are using forkprove, but should be fine with the standard prove tool.)

    Or in an individual test:

    use Time::Limit "30"; # quote marks around the number are required!

    Update: actually it works OK with forkprove, if you get Time::Limit to send its kill signal to the process group:

    forkprove -MTime::Limit=-group,60 -Ilib /path/to/tests

      Thanks, that is just what I need.

      I will add an overall time-out of half an hour or so to the overall prove run, and then encourage other developers to add a more conservative time-out to each unit test, so that it times out more quickly if there is a bug.

Re: Timeout on prove
by perlfan (Vicar) on Jul 02, 2014 at 23:53 UTC
    Wrap whatever thing you're doing that times out in a alarm and eval block.

    You can do one of 2 things (or both):

    a. BAIL_OUT, or

    b. check $@ outside of the eval with an ok.

    In some case, you may actually want it to time out.

    However, that you're timing tells me that what is timing out should probably mocked. It also tells me that you may want to have a more robust wrapper around whatever it is that is timing out - because it may time out when run for real. Throwing Exception::Class is always good - and you can even test that you handle things like timeouts properly using Test::Exception.

    A quick and easy way to mock whatever it is that is timing out is to use a typeglob:

    # mock returned data *Package::method = sub { return 'some expected data' };
    Similarly, you can force a timeout:
    # mock a timeout, but wrap this in an eval/alarm my $a_long_time = 900; # seconds *Package::method = sub { sleep $a_long_time; return; };
    Or just straight up die throw an exception (i.e., from Exception::Class):
    # mock a die from timeout, but wrap this in an eval/alarm *Package::method = sub { die "Timed out!\n"; };
    # mock a timeout exception, but wrap this in an eval/alarm *Package::method = sub { throw My::Exception::Timeout; };

      Thanks, but the issue is not caused by slow connections to external resources. We already mock those. The issue was caused by a bug in normal perl code that had no external dependencies.

      What happened, is that the unit test contained code to set-up the initial test data, but that setup code was broken by a change elsewhere in the code base. A bit like:

      my @objects = map { My::Object->new() } 1..100; # Give a random 10 object the bar property, skip any where foo is set. for( 1..10 ) { my $object = $objects[ int rand 100 ]; redo if $object->is_foo(); $object->is_bar(1); } # Use the object array in the unit tests.

      The problem was that someone else changed the default constructor for My::Object, so that foo was always set to true, so the while loop above looped forever.

      Nothing to do with external resources, just quick and dirty test code broken by an API change.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://1091994]
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others chanting in the Monastery: (6)
As of 2024-03-28 20:06 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found