local *STDERR; doesn't fix that

That's because system calls some c function which doesn't know about *main::STDERR, and which takes file descriptor 2 to mean STDERR.

print "local *STDERR$/";
{
   local *STDERR;
   system $^X, qw[ -le warn(6) ];
}
print "close STDERR$/";
{
   close STDERR;
   system $^X, qw[ -le warn(6) ];
}
__END__
local *STDERR
6 at -e line 1.
close STDERR
[download]

Is there some way I can stop that without having to patch Test...

Don't patch Test::Harness::Straps , just subclass it and use File::Spec->devnull()