To start with an overview: I have a big file of Western-encoded messages, you could call it some sorta non-standard blog, marked-up with HTML. Am now trying to clean up things, storing the messages as XML (more specific: as RSS 2.0), and displaying them as HTML. The encoding should change to UTF-8.

Most of the things work, just the UTF-8 encoding of special entities for the XML drives me nuts. I have been deploying the XML::RSS Perl module, which might or might not be a good idea afterOverview: I have a big file of Western-encoded messages, you could call it some sorta non-standard blog, marked-up with HTML. Am now trying to clean up things, storing the messages as XML (more specific: as RSS 2.0), and displaying them as HTML. The encoding should change to UTF-8.

Most of the things work, just the UTF-8 encoding of special entities for the XML drives me nuts. I have been deploying the XML::RSS Perl module, which might or might not be a good idea after testing it. E.g. sometimes the encode_output switch is being ignored depending on which server I execute the script.

It also seems XML::RSS does not correctly support the common way of encoding/decoding UTF-8 entities. In those cases where the mentioned "encode_output" of XML::RSS does work it produces something like this for the lower-case 'a' with two dots on top: testing it. E.g. sometimes the encode_output switch is being ignored depending on which server I execute the script.

It also seems XML::RSS does not correctly support the common way of encoding/decoding UTF-8 entities. In those cases where the mentioned "encode_output" of XML::RSS does work it produces something like this for the lower-case 'a' with two dots on top:

ä

When XML::RSS reads in entities like this it gets correctly decoded. But some common RSS readers are not swallowing it. It _seems_ that the encoding for the above example should have been:

쎤

When XML::RSS reads entities like this, something goes wrong and I see funny characters as result. After some frustration I had written the encoding myself like the second version, which solves the encoding part but XML::RSS does not like it.

    My questions are:
  1. Are both ways above correct when encoding UTF-8 in XML?
  2. Is using XML::RSS a bad idea? Any alternatives?
  3. How to best encode those entities in HTML for output?
  4. Side topic: Could it be that RSS readers better support decimal encoding, e.g. 쎤 than the hexadecimal equivalent 쎤 ?

Would be really glad if someone could help with any part of it

Jot

20080909 Janitored by Corion: Added closing list tag, as per Writeup Formatting Tips

In reply to UTF-8 entities in XML/HTML? by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.