From my reading of Advanced Perl Programming 2nd ed., best practice for utf8 handling is to mark utf8 data a such as it comes into your program and mark file handles as being utf8 when they are expected to read or write utf8.

I am using CGI.pm to process a form POST that sometimes contains utf8 data (which is urlencoded). The problem is that data retrieved by $q->param('foo') does not have the utf8 flag set when it contains utf8 data.

If I call $q->charset('utf-8') then the data is marked as utf8. But why should I have to do this?! Shouldn't the client tell the server what kind of data is being sent and CGI.pm do the right thing automatically?

Perhaps part of the problem is that I don't understand the content-type x-www-form-urlencoded. To me that content-type says nothing about what type of characterset the urlencoded data is representing.

Finally, is there any harm in calling $q->charset('utf-8') in all cases? I hate to do it but it seems necessary.

Thank you for any help.

In reply to Understanding CGI.pm and UTF-8 handling by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.