Re^2: Composite Charset Data to UTF8?

ok, thats like my first approach:

use utf8;
use open ':std', ':encoding(UTF-8)';
use open IO => ':encoding(UTF-8)';
[download]

but ok.. internal error like this:
utf8 "\xA9" does not map to Unicode at /usr/local/share/perl/5.14.2/XML/Tidy.pm line 780.
utf8 "\xAE" does not map to Unicode at /usr/local/share/perl/5.14.2/XML/Tidy.pm line 782.

anyway that is not the really part of the problem.. anybody got a quick solution to test a file for a constant charset? e.g. true/false for file eq utf8 or not?! can i say that the file is utf after utf8::decode($_) or die "Input is not valid UTF-8";just to say there are more then one charsets in the file or not??? or is it part of the problem?!

kindly perlig

$perlig =~ s/pec/cep/g if 'errors expected';

Comment on Re^2: Composite Charset Data to UTF8? Select or Download Code

Replies are listed 'Best First'.
Re^3: Composite Charset Data to UTF8? by Corion (Patriarch) on Jun 19, 2013 at 12:07 UTC
Have a look at the encoding rules of UTF-8. A valid UTF-8 sequence starts either with `0b0xxxxxxx` or with `0b11xxxxxx`. So any octet starting with `0xb10xxxxxx` is invalid UTF-8: `> perl -wle "print sprintf '%08b', $_ for (0xa9,0xae)" 10101001 10101110` [download] An untested easy check could be to match your string against `/[\x80-\xBF]/`, which are the hex representations of the bit patterns we've identified: `perl -wle "print sprintf '%08b - %02x', $_,$_ for (0b10000000,0b101111 +11)" 10000000 - 80 10111111 - bf` [download]	[reply] [d/l] [select]

Replies are listed 'Best First'.

Re^3: Composite Charset Data to UTF8?
by Corion (Patriarch) on Jun 19, 2013 at 12:07 UTC

Have a look at the encoding rules of UTF-8.

A valid UTF-8 sequence starts either with 0b0xxxxxxx or with 0b11xxxxxx. So any octet starting with 0xb10xxxxxx is invalid UTF-8:

> perl -wle "print sprintf '%08b', $_ for (0xa9,0xae)"
10101001
10101110
[download]

An untested easy check could be to match your string against /[\x80-\xBF]/, which are the hex representations of the bit patterns we've identified:

perl -wle "print sprintf '%08b - %02x', $_,$_ for (0b10000000,0b101111
+11)"
10000000 - 80
10111111 - bf
[download]

[reply]
[d/l]
[select]