in reply to Re^4: Decoding Russian text
in thread Decoding Russian text

Encode::Guess says it can't distinguish between single-byte encodings.

I'll definitely start mentioning Encode::Detect which I hadn't heard of before.

Replies are listed 'Best First'.
Re^6: Decoding Russian text
by Jim (Curate) on Jul 14, 2011 at 19:27 UTC

    I thought to mention the Perl encoding detection modules specifically because your script probably doesn't distinguish between single-byte encodings such as KOI8-R and Windows-1251. (Or does it?) vit seems to be asking for automated encoding detection in Perl that can at least distinguish between the common Cyrillic legacy encodings, which are single-byte encodings. In the general case, this is a sticky wicket.

    Needless to say, character encoding guessing is guesswork. And disambiguating characters in single-byte encodings is much more guess-y than it is in multi-byte encodings.

    (For fun, read Bush hid the facts.)

    Jim

      I thought to mention the Perl encoding detection modules specifically because your script probably doesn't distinguish between single-byte encodings such as KOI8-R and Windows-1251.

      It does, but it relies on human inspection instead of mechanical inspection. Both approaches are useful, but they have different merits and different use cases.