Re^3: Perl detect utf8, iso-8859-1 encoding

I'm guessing the bug is in Text::Unaccent, but it's directly using the iconv C library, so I can't easily say for sure.

However, maybe this can work:

use strict;
use feature qw(unicode_strings say);
use Unicode::Normalize 'NFD';

my $author = "Sch\x{f6}\x{f6}ttl";

$author = NFD $author;
$author =~ s/\p{Combining_Diacritical_Marks}//g;
say $author;
[download]

This doesn't include and decode() or encode() of the incoming/outgoing strings. Also, I think that this can also break in cases where there are multiple combining characters.

Comment on Re^3: Perl detect utf8, iso-8859-1 encoding Download Code

Replies are listed 'Best First'.
Re^4: Perl detect utf8, iso-8859-1 encoding by swiftlet (Acolyte) on Jul 24, 2020 at 15:07 UTC
thanks for your code, sorry I didn't explain the problem clear enough. The input could be encoded in iso-8859-1 \x{f6}\x{f6}, or, maybe in utf-8, \x{c3}\x{b6}, I have to find out what is the charset first. Encode::Detect::Detector is the one I am using to find out what is the charset of the string, utf-8 or iso-8859-1, `the logic is like: $charset = = Encode::Detect::Detector::detect($input); if($charset eq 'UTF-8'){ # do NFC ... }elsif($charset eq 'iso-8859-1'){ # do NFD ... }` [download] Text::Unaccent unac_string($charset, $str) in my case. Text::Unaccent is working well if Detector can find it the correct code, it failed if Detector failed, of course, no charset. Encode::Detect::Detector normally working well, but failed if input = \x{f6}\x{f6}.	[reply] [d/l]
Re^5: Perl detect utf8, iso-8859-1 encoding by jeffenstein (Hermit) on Jul 24, 2020 at 20:06 UTC
Ah, oops. My fault there. You did explain it well, but I wasn't paying enough attention. I'll leave as is, just in case someone finds it useful one day	[reply]