Re: Perl detect utf8, iso-8859-1 encoding

Since no one has said it: detecting for encoding is always broken and should only ever be used as a last resort. If you have any way to either be sure of the encoding or enforce it up front, use it. If at all possible, look further up the chain for a way to do it correctly. Detection is incorrect, even if useful when there are no other options.

Comment on Re: Perl detect utf8, iso-8859-1 encoding

Replies are listed 'Best First'.
Re^2: Perl detect utf8, iso-8859-1 encoding by swiftlet (Acolyte) on Jul 25, 2020 at 00:14 UTC
I have charset="UTF-8" in "Content-Type", "<meta>" tag, and accept-charset="UTF-8", anything else can I enforce it? Since our old iso-8859-1 links may have been saved by users or indexed by search engines, I can't find another way to get the search backward-compatible without detection, any suggestions?	[reply]
Re^3: Perl detect utf8, iso-8859-1 encoding by Your Mother (Archbishop) on Jul 25, 2020 at 00:23 UTC
That sounds like a good first link in the chain. If you are receiving all your input through forms from UTF-8 declared pages, then you are only receiving UTF-8 data and you can treat it that way, ignoring all other encodings and definitely have no need to guess. If that’s all true and you’re having problems, probably you are not decoding correctly from the first step to the next processing steps. We’d need more info about the full processing chain to guide you there. This is overwhelming—and overkill because most basic processing chains don’t need to consider most of it—but it is the Rosetta Stone for the issues: Why does modern Perl avoid UTF-8 by default?	[reply]