You never know when and by what are you going to get caught. Recently we had a big problem with one client. We are getting some data from another company in XML. It's all english text so we did not pay any attention to encoding (UTF8). (Well the import was in ASP+VBScript so there was not much we could do.)

Everything was fine for most clients, except one. Because they were using MS Word to prepare the data that they then pasted into the other company's system. And MS Word's "Autoformat As You Write" screwed us up. It converted double and single quotes to some "smart quotes" and nice apostrophes. And these were 8bit chars and were encoded as 3 bytes in UTF8.

Long story short ... I did not find any reasonable way to fix the issue in VBScript so I ended up reimplementing the import in Perl ;-)

Jenda
Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.
   -- Rick Osborne

Edit by castaway: Closed small tag in signature

In reply to Re: Programmers, script languages, and Unicode by Jenda
in thread Programmers, script languages, and Unicode by dbwiz

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.