in reply to UTF-8 or iso-8859-1 input to CGI.pm

This is a good question and the answer is non-trivial to comprehend.

I recommend taking a look at this page, which is a mirror of a (now-defunct but highly useful) original for background: http://niwo.mnsys.org/saved/~flavell/charset/form-i18n.html

In short, you never *really* know exactly what you're getting back from a browser, and the very best belt-and-suspenders thing you can do is to have a "magic cookie" hidden form field on each form that contains a smattering of odd characters with an unambiguous representation in each encoding you suspect. That is, you put a bunch of CJK and other stuff, output as HTML entities, into your form field, and you use a heuristic based upon what the browser sends back to determine what encoding the browser has decided to use.

You can also look at Encode::Guess, which makes some informed guesses based on the content, but doesn't require making the round-trip the way the form field technique does.

But what most people do, and what's probably most tractable unless you're trying to solve a problem for a large, diverse user-base, is to figure out what the top 3-4 user-agents you serve do, and just bank on that...

  • Comment on Re: UTF-8 or iso-8859-1 input to CGI.pm