Re^2: Perl detect utf8, iso-8859-1 encoding

simply checking if the input is valid UTF-8 and assuming ISO-Latin-1 if not

Thanks! This is a good idea, but how could I find out if the input is a valid utf-8 or not? Both utf8::valid and utf8::is_utf8 are not working well in my examples

Comment on Re^2: Perl detect utf8, iso-8859-1 encoding

Replies are listed 'Best First'.
Re^3: Perl detect utf8, iso-8859-1 encoding by haj (Vicar) on Jul 25, 2020 at 08:50 UTC
To check whether data are valid UTF-8 is rather straightforward. Here's the example, slightly modified from the synopsis of Encode: `use Encode qw(decode encode); $characters = decode('UTF-8', $octets, Encode::FB_CROAK \| Encode::LEAVE_SRC);` [download] This code will `die` if there are invalid data, so you would wrap it into the exception handler of your choice, plain `eval` and Try::Tiny seem to be popular. BTW: as jcb already indicated, chances are excellent that if data pass as UTF-8, they actually are UTF-8. All bytes of multibyte characters in valid UTF-8 strings are in the range `\x80` to `\xFF`, and in particular the bytes 2-4 are in the range `\x80-\xBF`. You just can't build readable text from characters in that range in any of the ISO-8859-* encodings, and about half of that range are "unprintable" control characters from ISO/IEC 6429.	[reply] [d/l]

Replies are listed 'Best First'.

Re^3: Perl detect utf8, iso-8859-1 encoding
by haj (Vicar) on Jul 25, 2020 at 08:50 UTC

Encode

use Encode qw(decode encode);
$characters = decode('UTF-8', $octets, 
                     Encode::FB_CROAK | Encode::LEAVE_SRC);
[download]

This code will die if there are invalid data, so you would wrap it into the exception handler of your choice, plain eval and Try::Tiny seem to be popular.

BTW: as jcb already indicated, chances are excellent that if data pass as UTF-8, they actually are UTF-8. All bytes of multibyte characters in valid UTF-8 strings are in the range \x80 to \xFF, and in particular the bytes 2-4 are in the range \x80-\xBF. You just can't build readable text from characters in that range in any of the ISO-8859-* encodings, and about half of that range are "unprintable" control characters from ISO/IEC 6429.

[reply]
[d/l]