This is pretty timely for me -- I've recently been setting up an environment and writing tests for the next version of OpenInteract. Most of the tests are fairly simple, but they just don't work outside of a running application environment. And it takes a little while to setup that environment -- about 10-20 totally unoptimized seconds right now.
I guess I had the unrecognized assumption that running the full test suite was a good thing and most people -- that is, the default for whatever tests I write -- wouldn't mind running them.
I'll have to mull this over for a while... I see the point but I'm also greedy with the mass of testbeds out there available to test on. Like adrianh said, it's one of the powerful (but underrecognized) benefits of CPAN and the infrastructure built up around it.