in reply to Re^2: How to handle encodings?
in thread How to handle encodings?

A typical place where Encode::Guess falls down (through no fault of its own) is in differentiating one variant of iso-8859 from another.

Who's to say if chr(250) is "Č" (ISO-8859-2) or "Θ" (ISO-8859-7)?

Without prior knowledge, you're up the creek without a paddle. So I agree wholeheartedly with Moritz's suggestion of converting everything to UTF8, while you still know what encoding it is in.

(graff - I know you're too wise a monk to have been suggesting otherwise, but I wanted to provide a simple example of just how limited Encode::Guess can be.)

Clint

Replies are listed 'Best First'.
Re^4: How to handle encodings?
by Narveson (Chaplain) on Mar 07, 2009 at 22:28 UTC
    Who's to say if chr(250) is "Č" (ISO-8859-2) or "Θ" (ISO-8859-7)?

    Any moderately experienced human eye, given a paragraph of context. Have you ever played the game of pulling down the Character Encoding menu in your Web Browser and trying the same page in different encodings?

    This is not a dissent from moritz's and clinton's advice. By all means heed any available declarations.

    But in the absence of a declaration, the right encoding is humanly guessable, even when the alternatives are different members of the ISO-8859 series. And if it's humanly guessable, then a CPAN module should in principle be able to do it too.

    I don't know what algorithm Encode::Guess uses. My own algorithm (untested) would be

    • submit a phrase to Google as a search term;
    • accept Google's proposed spelling correction.

[reply]

In Section Seekers of Perl Wisdom