I have to admit this is something I'd never do. If there's a test I want it run, otherwise I wouldn't have written it. I'd much rather have somebody wait a few minutes than have a module installed with test failures. That way lies madness :-)

There's also the overhead of printing the test output which schwern has said he'll fix at some point. For modules like M::BI that may be significant.