Great idea! Here's a quick and dirty implementation:

package Module::Build::SillyPodTests;

use SUPER;
use base 'Module::Build';

use Module::Build;

sub ACTION_disttest
{
    my $self = shift;
    local $ENV{PERL_TESTPOD} = 1;
    super();
}

sub find_test_files
{
    my $self  = shift;
    my $tests = super();

    return $tests if $ENV{PERL_TESTPOD};
    return [ grep { $_ !~ /\bpod.*\.t\z/ } @$tests ];
}

1;
[download]

Use this in place of Module::Build in your Build.PL file and the only suck is that you still have to distribute the why-make-users-run-them POD testing files.

Edited somewhat.