Fast coverage module or hack

vsespb has asked for the wisdom of the Perl Monks concerning the following question:

Hi.

I have the following system running: It executes tests for large project on CI server with code coverage (with Devel::Cover ) turned on, after that it creates map telling which test executes which module. I.e. if, when running test1.t and modules Mod1.pm and Mod2.pm have non-zero statement coverage, it add record to map: test1.t(Mod1.pm, Mod2.pm).

Next, that map is used to determine which tests to be ran in next CI run. I.e. if I have record test1.t(Mod1.pm, Mod2.pm), that means when Mod1.pm modified, then test1.t should be ran, and if Mod3.pm modified test1.t should NOT be run. This way I run fewer tests each time and save lots of time running tests.

Question is: Devel::Cover is too slow, I need to find thing which is faster than Devel::Cover, but still fit my need. It can be as simple as:

Devel::Cover hack which stop collecting stats (and start work faster) for the file after determining that the file has non-zero coverage.
Some other module to get info that "some" code is executed from module X during run (except code in "BEGIN" blocks).

Any ideas?

Comment on Fast coverage module or hack

Replies are listed 'Best First'.
Re: Fast coverage module or hack by tobyink (Canon) on Dec 02, 2019 at 07:50 UTC
First idea: if you load modules on demand — `require` at run-time instead of `use` at compile-time — then `%INC` will tell you what modules each test script has tested. Second idea: the map probably doesn't change that often, so you don't need to update it every check-in. Check when it was last updated, and only run your test suite with Devel::Cover on to update the map if the map is stale by more than X hours. (X might be 6?) After a major refactoring, you might want to manually re-trigger the map to be regenerated. toby döt ink	[reply] [d/l] [select]
Re^2: Fast coverage module or hack by vsespb (Chaplain) on Dec 02, 2019 at 08:32 UTC
First idea - no. we use "use" and compile time to load all stuff before "fork" webserver and to save memory with COW, so every module "use" maybe 80% of other modules. Second idea - yes we already do this. We run map collection once per day (before Devel::Cover is so slow for us). But when map changes, it actually can break things for rest of day. So we run map collection once per day, but also do git-diff to the commit where map collection was made, then there is no possibility of use of wrong map then, but diff gets bigger during the day. So we're looking how to speed up coverage module.	[reply]
Re^3: Fast coverage module or hack by tobyink (Canon) on Dec 02, 2019 at 12:29 UTC
How slow are we talking about? 10 minutes? An hour? 6 hours? Would doing the Devel::Cover run overnight instead of the day help? toby döt ink	[reply]
Re^4: Fast coverage module or hack by vsespb (Chaplain) on Dec 02, 2019 at 12:36 UTC
Re^3: Fast coverage module or hack by tobyink (Canon) on Dec 02, 2019 at 12:39 UTC
I think I have an idea for a solution but it would require a one-line change (a `use` statement added) to each module. Does that sound acceptable? My solution will need me to do a little work to figure out the details, but it seems like an interesting task to work on, and it might help me with one or two of my own projects. Edit: actually probably don't need a change to each module. Edit again: actually do. toby döt ink	[reply] [d/l]
Re^4: Fast coverage module or hack by vsespb (Chaplain) on Dec 02, 2019 at 14:39 UTC
Re^5: Fast coverage module or hack by tobyink (Canon) on Dec 02, 2019 at 16:03 UTC
Some notes below your chosen depth have not been shown here
Re: Fast coverage module or hack by Anonymous Monk on Dec 02, 2019 at 07:26 UTC
Any ideas? Stop testing ? Yup If Devel::Cover is too slow, you're using it way too much Just how fast can you write code anyway that Devel::Cover is too slow?	[reply]