Assuming perl thinks it's utf8 data to begin with, you can catch the "invalid" warnings before they happen
with something like (untested) Encode::_utf8_off($str) if !utf8::valid($str)
Update: Caveat programmer: don't ever use _utf8_off or _utf8_on except where you know perl has the utf8 flag wrong.