This could get dicey, because there are lots of ways for a given byte sequence to violate the various constraints of utf8. When in doubt about how bad the data might be, the best first step would generally be one that just traps and reports the problem -- something like this:
my $untrusted = <>; # (or whatever you use to fetch it) my $test_utf8; eval "\$test_utf8 = decode( 'utf8', \$untrusted, Encode::FB_CROAK )"; if ( $@ ) { warn "Input data is not valid utf8: $@"; # take other appropriate action(s)... }
It might be worth while to run a script that does just this on the input data, before you pass the data into any other script to actually do something with it, just to see whether the problems are infrequent or rampant. Sometimes they are fairly easy to diagnose, but you may need to be able to track through a hex dump of the relevant bytes and be able to understand both the intention and the nature of how that differs from the reality.

I don't know if it'll help, but a prior discussion about a similar kind of problem came up here: Guess between UTF8 and Latin1/ISO-8859-1


In reply to Re: UTF-8 Validation by graff
in thread UTF-8 Validation by daviddhall

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.