doulbe/triple/multi encoded utf8

fert has asked for the wisdom of the Perl Monks concerning the following question:

I've been having some trouble with a certain web API from a certain large online marketplace site... They are returning multiple results that seem to be the same just different iterations of applying utf8 encoding to the same string. Ex:

Some CompanyÃÂ®
Some CompanyÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂ® 
Some CompanyÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂ®
[download]

I'm pretty sure the intention in this example is the ® symbol - a registered trademark type symbol. Somehow on the companies end this data is getting screwed up, and they are apparently working on it - though that might mean a few months of no fix, and I have historical data that I need to use in the mean time. What I'm looking for is an equivalent way to do what the "utf8_decode" function in PHP does, but in Perl. A simple example in PHP would be something like this, that does what I want:

$prev = $value;
for($i=0;$i<999;$i++) {
    $value = utf8_decode($value);
    if ($prev == $value) {
        break;
    }
    $prev = $value;
    print "[$i]$value\n";
}
[download]

I've been trying to get Encode::encode and Encode::decode to work here, but they don't seem to be able to actually unravel this mess. With my PHP hack script I can decode it to this:

Some CompanyÃÂ®
[0]Some CompanyÂ®
[1]Some Company®
------------------
Some CompanyÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂ® 
[0]Some CompanyÃÂÃÂÃÂÃÂ® 
[1]Some CompanyÃÂÃÂ® 
[2]Some CompanyÃÂ® 
[3]Some CompanyÂ® 
[4]Some Company® 
------------------
Some CompanyÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂ® 
[0]Some CompanyÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂ® 
[1]Some CompanyÃÂÃÂÃÂÃÂ® 
[2]Some CompanyÃÂÃÂ® 
[3]Some CompanyÃÂ® 
[4]Some CompanyÂ® 
[5]Some Company®
[download]

I would probably then want to re-encode the final one back into utf8 and proceed from there. I know this is very hacky, but I need a workaround until they fix the data source. Any ideas?

Comment on doulbe/triple/multi encoded utf8 Select or Download Code

Replies are listed 'Best First'.
Re: doulbe/triple/multi encoded utf8 by Corion (Patriarch) on Jan 07, 2010 at 17:29 UTC
This sounds like an almost perfect case for Encode::Repair, just released by moritz.	[reply]
Re^2: doulbe/triple/multi encoded utf8 by fert (Acolyte) on Jan 07, 2010 at 18:16 UTC
great, thanks, exactly what I was looking for, I will check that out right away	[reply]
Re: doulbe/triple/multi encoded utf8 by ikegami (Patriarch) on Jan 07, 2010 at 18:08 UTC
`sub multidecode_utf8 { my $s = shift; 1 while $s =~ tr/\x80-\xFF// && utf8::decode($s); return $s; }` [download] Tested with: `use strict; use warnings; use Test::More tests => 12; for my $orig ( "abc", "\xC9ric", "\x{2660}\x{2661}\x{2662}\x{2663}", ) { my $str = $orig; for (0..3) { is multidecode_utf8($str), $orig; utf8::encode($str); } }` [download]	[reply] [d/l] [select]