in reply to Confusing UTF-8 bug in CGI-script

The problem is that you are decoding STDIN (via use open), and STDIN is used to transfer something that isn't text. The solution is to add 

binmode(STDIN);

[download]

The problem in detail

CGI (the protocol) uses STDIN to pass on the document sent in POST requests. In this case, it's an application/x-www-form-urlencoded document.

Then, CGI (the module) parses that document and decodes the extracted text components.

Encode (used by CGI) detects (correctly guesses) that something's wrong and fails with 

Cannot decode string with wide characters

[download]

Don't decode non-text. Other processing (decoding of "%" escapes) needs to be done before you have the text that needs to be decoded.

Replies are listed 'Best First'.
Re^2: Confusing UTF-8 bug in CGI-script
by wanradt (Scribe) on Feb 01, 2011 at 20:14 UTC
    STDIN is used to transfer something that isn't text.

    What you mean: isn't text. What else? And how then transfer the text and make perl to understand it is UTF-8 encoded?

    Strange thing: i have full site running years in UTF-8, every CGI-script has this "use open ':std' => ':encoding(UTF-8)';" at beginning (pretty much the same init block as in this example above), because without it i just did not get anything to work... Now i copied it to another project, stripped down to skeleton and it does not work anymore... It is too mysterious to me.

    Nõnda, WK
[reply]

      What you mean: isn't text.

      The text you typed into your browser is transformed by it as follows:

      1. It is encoded using the proper character encoding.
      2. Some of it is encoded using percent encoding.
      3. The resulting string is joined to others to form a application/x-www-urlencoded document.

      That leaves you something that's no longer your text. The proper inverse of that is:

      1. Split the form data into its components.
      2. Remove any percent encoding.
      3. Remove the character encoding.

      You're adding an additional step:

      1. Remove the character encoding. (XXX)
      2. Split the form data into its components.
      3. Remove any percent encoding.
      4. Remove the character encoding.

      The fourth step notices something is odd and throws an error.

      And how then transfer the text and make perl to understand it is UTF-8 encoded?

      That's what the «-utf8» in «use CGI qw(:all -utf8);» does. "This makes CGI.pm treat all parameters as UTF-8 strings" by passing them to decode.

[reply]
[d/l]
[select]
        That's what the «-utf8» in «use CGI qw(:all -utf8);» does.

        And without "-utf8" i can't use CGI properly because then are UTF-8 encoded GET-paramaters treated wrong? I mean, STDIN i can fix with "use open ..."

        I'm gonna finally get to somewhere... Thank you!

        Nõnda, WK
[reply]

      Huh, at least i found the difference between working production code and script here: in production i initialize CGI-object (inside BEGIN-block) before asking use open to decode STDIN. Seems, it is the significant difference. Still i hope, you could explain, why input from STDIN is not text.

      Nõnda, WK
[reply]
        The CGI object only reads from STDIN during initialisation. Anything you do to STDIN later won't affect the CGI object.
[reply]

In Section Seekers of Perl Wisdom