Intercepting DynaLoader::dl_load_file(...) to verify the binary bits of XS modules would make this a lot more robust. I'd be more worried about a virus or something being written to inject code in a lib/dll than into a pure Perl module anyway.

BEGIN {
    require DynaLoader;
    # no strict 'refs';
    no warnings 'redefine';
    my $keep = \&DynaLoader::dl_load_file;
    *DynaLoader::dl_load_file = sub {
        my ( $path, $flags ) = @_;
        warn "We should check '$path' here";
        &$keep(@_);
    };
}

# Random XS based core modules
use Cwd;
use Fcntl;
use Digest::MD5;
[download]

I guess you'll need to think about FFI loaded libraries eventually.