yet when someone fails to handle it, they don't move on until it's fixed.

I don't think I understand that statement? Do you mean: When programmers encounter a problem with handling utf-16, they neither try to correct it, nor fail loudly?

From my perspective, the first problem is that when you read a file, and there is simply no way to know what it contains. Unless you know upfront what is in the file, there is no sensible mechanism for deciding how you should decode (or encode?) the contents of that file. Guess wrong and you produce nonsense. Know wrong and you produce nonsense. Download text from the web and if the webmaster has--through laziness, incompetence or maliciousness--miscoded the mime-type and you're stuffed.

The second problem is, as I said, variable length encoding. With a fixed length encoding, if you want to fetch the 3 millionth character of your data, you add 3e6 to your base and fetch it. With variable length, you have to inspect every single one of the intervening 6 million to 12 million bytes.

Imagine if the postman had to walk the length of Yonge Street inspecting each house name in order to deliver a letter :) (Or IP traffic if...)

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
"I'd rather go naked than blow up my ass"

In reply to Re^6: Does String::LCSS work? by BrowserUk
in thread Does String::LCSS work? by BrowserUk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.