Perl seems unable to properly handle the slash-x style octets that represent characters in the upper-ascii range or beyond--or perhaps, as is likely, I just don't know what I'm doing! With that in mind, someone can probably spot my error. Here's the issue.

I need to convert files which have stored Latin1-style characters in a (to me) non-standard UTF-8 format to "standard" UTF-8 characters (the actual characters instead of representations of them).

THIS:

\xc3\xa4    <span class="sy">\xc3\xa4</span>, <span class="sy">\xc3\x8
+4</span><span class="posg pos">Substantiv, Neutrum, das</span><span c
+lass="vg v"> \xc3\x84  \xc9\x9b\xcb\x90 das \xc3\xa4; Genitiv: des \x
+c3\xa4 (umgangssprachlich: -s), \xc3\xa4 (umgangssprachlich: -s) </sp
+an>

[download]

Should be converted to THIS:

ä    <span class="sy">ä</span>, <span class="sy">Ä</span><span class="
+posg pos">Substantiv, Neutrum, das</span><span class="vg v"> Ä  &#603
+;&#720; das ä; Genitiv: des ä (umgangssprachlich: -s), ä (umgangsspra
+chlich: -s) </span>

[download]

I've tried everything I can think of. I am able to correctly convert it in a browser by using the following sequence:

use Encode;
use utf8;

binmode STDOUT, ':utf8';

print "Content-Type:text/html; charset=utf-8\n";
print "Content-Language: utf8;\n\n";

my $ucode = qq|\xc3\xa4    <span class="sy">\xc3\xa4</span>, <span cla
+ss="sy">\xc3\x84</span><span class="posg pos">Substantiv, Neutrum, da
+s</span><span class="vg v"> \xc3\x84  \xc9\x9b\xcb\x90 das \xc3\xa4; 
+Genitiv: des \xc3\xa4 (umgangssprachlich: -s), \xc3\xa4 (umgangssprac
+hlich: -s) </span>|;

my $newcode = decode('utf8', $ucode);

print "<p>$newcode</p>\n";

[download]

But if I read the line from a file, perform this conversion, and then push that line into an array and print it to a file, I still end up with the slash-x characters that I wanted to abolish! This causes me to think that Perl had not worked the magic at all--it all came from my browser. I hate to think that FireFox handles text encodings more powerfully than Perl is capable of!

What must I do?

Blessings,

~Polyglot~

In reply to Perl's encoding versus UTF8 octets by Polyglot

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.