A good compilation, to which I'd like to add (or maybe in the cases 6 and 7 just expand on your fourth entry):

5) @ARGV and %ENV. utf8::all does convert @ARGV but does not touch %ENV. Good luck when you're on Windows where per default the terminal doesn't use UTF-8 encoding.

6) Database fields. Encoding of these is often defined outside of the Perl world. The driver docs should tell you how to handle encoding.

7) Evaluating binary data in your program: Unzipping compressed data, decrypting secret stuff, and parsing ASN.1 or image metadata may all return (encoded) texts.

I'm also not too happy with using Devel::Peek for debugging encoding issues. It provides too much useless information, sometimes misleading, and is difficult to read (like PV = 0x5629d24aaa30 "\303\244"\0 [UTF8 "\x{e4}"] versus PV = 0x5629d2414060 "\344"\0). I'd rather write suspicious strings to a file, using UTF-8 encoding, and examine this file with an editor which is capable of UTF-8 and hex display.

I'm also using some regular expressions in debugging:

my $utf8_decodable_regex =
    qr/[\xC0-\xDF][\x80-\xBF]    | # 2 bytes unicode char
       [\xE0-\xEF][\x80-\xBF]{2} | # 3 bytes unicode char
       [\xF0-\xFF][\x80-\xBF]{3}/x;

sub contains_decodable_utf8 { $_[0] =~ /$utf8_decodable_regex/; }

sub is_utf8_decodable {
    $_[0] =~ /\A($utf8_decodable_regex|[[:ascii:]])*\z/;
}
[download]

• If contains_decodable_utf8($string) is false, then you should be fine.
• If is_utf8_decodable($string) is true, then you can (and should) decode the string.
• If contains_decodable_utf8($string) is true but is_utf8_decodable($string) is false, then you either have binary data (which might be just fine) or you have already mixed up encodings. Go back in your code and check what you did to $string before.

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.