- How do you handle this?

I keep everything in UTF-8, since it's something that's universal and that's understood by nearly every program

- Is it a good idea to convert incoming data to Perls internal format when processing it, and do vice versa when printing/storing processed data?

Yes, it's the way to go IMHO. You can use IO layers and Encode to do it for you.

- Does the format of the file containing the Perl code itself matter?

If there are string constants in that file, and you concatenate them to the data, it does matter. So you should decode these string constants (or keep the perl files in UTF-8, and use utf8; which does the decoding for you).

I've looked att the Encoding::Guess module, is this an option to decide the format of the incoming data?

No. Guessing encoding is not reliable, and you should avoid it by all means. Make sure that all your interfaces have a way to specify the encoding.

My concluding question: What is the best way to deal with different character sets in a system?

Keep all data internally in a consistent format, and recode at the boundary between what you consider "internal" and "external". The internal encoding should be a Unicode encoding (like UTF-8, UTF-16{l,b}e) so that you won't have any information loss during recoding. Unicode aims for round-trip conversions between non-Unicode charsets and Unicode, and for all common encodings it pretty much succeeds.

In reply to Re: How to handle encodings? by moritz
in thread How to handle encodings? by DreamT

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.