I am working with documents that use iso8859-1-2-5 encoding and some nonstandard encoding and fonts that need to be mapped to utf8. To clean things up I simply want to do substitutions within the input strings. As you can see in the test case below this does not work 100% of the time. 
#perl -w
require 5.8.0;

use strict;
use utf8;
use Encode;

${^WIDE_SYSTEM_CALLS} = 1;

#no warnings 'utf8';

open STDOUT, "> STDOUT";
binmode(STDOUT, ":utf8");
open STDERR, "> STDERR";
binmode(STDERR, ":utf8");

my @a_lc_grave          = ('à', "\x{00E0}", 'a');
my @a_lc_diaeresis      = ('ä', "\x{00E4}", 'a');

my @acy                 = ('а', "\x{0430}", '');
my @dcy                 = ('д', "\x{0434}", '');

my $text = '';
$text = ${a_lc_grave[1] . ${a_lc_diaeresis[1]}};
Encode::encode_utf8($text);
&test("success");

$text = ${a_lc_diaeresis[1]} . ${a_lc_grave[1]};
Encode::encode_utf8($text);
&test("fail");

$text = ${a_lc_grave[1]} . ${a_lc_grave[1]};
Encode::encode_utf8($text);
&test("success");

$text = ${a_lc_diaeresis[1]} . ${a_lc_diaeresis[1]};
Encode::encode_utf8($text);
&test("success");

sub test ()
{
   print "-"x20, "\n";
   print $_[0], "\n";

   print "Before = ", unpack ("U*", ${text}), "\n\n";

   $text=~s/${a_lc_diaeresis[1]}/${dcy[1]}/g;
   $text=~s/${a_lc_grave[1]}/${acy[1]}/g;

   print "After = ", unpack ("U*", ${text}), "\n\n";
   print $text, "\n\n";
}

[download]

In reply to utf8 encoding bug? by zemplen

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.