None of my modules are large enough to have a massive test suite, but I imagine in a large test suite you're right that it might be possible to make some tests optional. I personally try not to require a ton of modules for testing (except in my author tests), i.e. the same policy as for the modules themselves (e.g. use Moo instead of Moose when possible and so on). Also, though I'm not sure at the moment how the different CPAN clients handle this, I believe that TEST_REQUIRES dependencies are the ones that don't need to be installed into the user's environment.
Update: In fact, as an example, because I really only need one functionality from Test::Exception Test::Fatal and Test::Warnings each, I now quite simply inline that:
sub exception (&) { eval { shift->(); 1 } ? undef : ($@ || die) }
sub warns (&) { my @w; { local $SIG{__WARN__} = sub { push @w, shift }
+; shift->() } @w }