Re^2: How to handle encodings?

I've looked att the Encoding::Guess module, is this an option to decide the format of the incoming data?
No. Guessing encoding is not reliable, and you should avoid it by all means. Make sure that all your interfaces have a way to specify the encoding.

I wouldn't be so harsh on Encode::Guess. It definitely can be useful when applied correctly to the right problems, and I think its man page does an okay job of saying what its strengths and weaknesses are.

I agree that using it as a "do-all" for every multi-encoding task would be wrong; ideally, all your inputs will provide some sort of declarative or unambiguous evidence about the encoding being used, but for inputs that don't provide that, you may need all the help you can get (doing "offline" research/investigation to understand the data) in order to figure out what encoding the data is using, and Encode::Guess can help in such cases.

Once you understand your data well enough, and you understand how Encode::Guess handles it, you may actually find it worthwhile to use the module in a production pipeline, to route data according to what the module can tell you about it (in the absence of any other information) -- but doing so without thorough testing would be a mistake.

Comment on Re^2: How to handle encodings?

Replies are listed 'Best First'.
Re^3: How to handle encodings? by clinton (Priest) on Mar 07, 2009 at 19:06 UTC
A typical place where Encode::Guess falls down (through no fault of its own) is in differentiating one variant of iso-8859 from another. Who's to say if `chr(250)` is "Č" (ISO-8859-2) or "Θ" (ISO-8859-7)? Without prior knowledge, you're up the creek without a paddle. So I agree wholeheartedly with Moritz's suggestion of converting everything to UTF8, while you still know what encoding it is in. (graff - I know you're too wise a monk to have been suggesting otherwise, but I wanted to provide a simple example of just how limited Encode::Guess can be.) Clint	[reply] [d/l]
Re^4: How to handle encodings? by Narveson (Chaplain) on Mar 07, 2009 at 22:28 UTC
Who's to say if chr(250) is "Č" (ISO-8859-2) or "Θ" (ISO-8859-7)? Any moderately experienced human eye, given a paragraph of context. Have you ever played the game of pulling down the Character Encoding menu in your Web Browser and trying the same page in different encodings? This is not a dissent from moritz's and clinton's advice. By all means heed any available declarations. But in the absence of a declaration, the right encoding is humanly guessable, even when the alternatives are different members of the ISO-8859 series. And if it's humanly guessable, then a CPAN module should in principle be able to do it too. I don't know what algorithm Encode::Guess uses. My own algorithm (untested) would be submit a phrase to Google as a search term; accept Google's proposed spelling correction.	[reply]