tr/// may be a bit faster than s///, so maybe (also untested)
    $blob =~ tr/\r\n\t\x20-\x7E//c or die "$filename is not ASCII";
(See perlop Quote-Like Operators for  tr/// and its  /c (complement) modifier.)

Update: Correction: The logical operator should be  and because we wish an exception to be thrown if any "non-ASCII" character is found, i.e., if the  tr///c count is non-zero:
    $blob =~ tr/\r\n\t\x20-\x7E//c and die "$filename is not ASCII";