The key bit for me here is "transforming HTML files into XML data sheets". The XML parser definition says that anything not in the ASCII range (unless specifically set in the header) is bad (and if you putatively have ASCII, but for some reason there are characters outside the normal ASCII range - say you're doing a test of a web crawler that hits a page with a Unicode snippet on it) --- and for an XML parser, "bad character" means "fall over dead".

I had this problem turning TAP output containing Unicode into JUnit XML; the solution was to translate any character outside of the printable ASCII range (except for newline and carriage return) to an &#xx; sequence, and the same for any embeded ", &, <, and > characters. The XML parser was then happy with that character set.

This is useful if you can't trust the encoding of the input to be right, since ASCII is kind of the "least common denominator" when it comes to character sets.

In reply to Re: stripping characters from html by pemungkah
in thread stripping characters from html by jonnyfolk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.