in reply to Encoding confusion with CGI forms

The clients send the data in a some encoding. You need to know the format. The key trick is to add a hidden field to the form. This field is also converted to the format that the clients use. Since you know how the hidden data look, you know the encoding of the formdata. Just convert them now into a common format and you are done.
Boris

Replies are listed 'Best First'.
Re^2: Encoding confusion with CGI forms
by Joost (Canon) on Oct 22, 2004 at 19:24 UTC
    Sounds like a nice trick. Especially since the official ways of handling request character encoding are not well supported and not very well thought out either (i.e. as far as I know a user-agent is only required to send charset information for multi-part forms)

    I'm just wondering about what string to use, though. There are dozens of encodings in use around the world, and you should be able (ideally) to recognize each one. Is there any "standard" way of doing this? (a CPAN module would be wonderful, ofcourse)

      I life in Germany so my string is only 'äöü' to distingush between ISO-8859-1, utf8 and unknown. But you can extend this to all your supported encodings, just find a char with different representations.
      Boris
      I really like this approach as well, I'm just having trouble coming up with a string that degrades in some predictable manner for different encodings.

      A perl module for this would be awesome, btw! It would be even more automagical if it were integrated into CGI.pm behind the scenes!

      Cheers,
      Troy

Re^2: Encoding confusion with CGI forms
by davistv (Acolyte) on Oct 22, 2004 at 20:50 UTC
    Hi Boris,

    The hidden form variable sounds like a great strategy, thank you for the tip. I've been playing with this concept today, but I'm not getting useful results so far.

    If I use the unicode smiley character as my hidden form field value (\x{263a}). I get back a character sequence like "☺", or \xe2\x98\xba in latin1. I can detect that with the regex /^\xe2\x98\xba$/, but even if the default encoding for Apache is UTF-8, the content-type charset in the resulting page is utf-8, the script is utf-8 (no BOM or perl gives an error like "(8)Exec format error: exec of '/var/www/cgi-bin/char2.cgi' failed"), and my browser is setting itself to UTF8 encoding as it should, and I copy and paste text from a document known to be in UTF-8, it's detected as latin 1 and not utf-8.

    So can I ask what string you use as a detection mechanism? And what are you using to match the mis-converted string in other encodings? I'm interested in Win1252 and Latin 1, if that makes any difference. My current source is below.

    Thank You,
    Troy

      Hi Troy,
      here is a untested example. Copy it to your cgi-bin/troy.pl. Without parameters it presenrs the submit form otherwise it shows the data everytime in utf8.
      The error in your script is that you confuse utf8 and unicode somewhere. Also the meta tags looks unnecessarily to me.
      PS: Consider to add readmore tags around your code on pm.
      #!/usr/bin/perl use strict; use Encode; use CGI; $|++; my $c = CGI->new; binmode( STDOUT, ":utf8" ); unless ( () = $c->param ) { print $c->header( -charset => 'utf-8' ), $c->start_html('Character conversion test'), qq{ <form action="/cgi-bin/troy.pl" method="post" enctype="multipart/f +orm-data"> <input type="hidden" name="enc_sniffer" value="\x{df}\x{20ac}"> <textarea name="textInput" rows="25" cols="72"></textarea><p> <input type="submit"> </form>}, $c->end_html; exit; } else { my $enc; my $hidden = $c->param('enc_sniffer'); { use bytes; $enc = ( $hidden eq "\xc3\x9f\xe2\x82\xac" ) ? 'utf8' : ( $hidden eq "\xdf\x80" ) ? 'cp1252' : 'iso-8859-1'; } # decode all param fields here my $qtext = decode( $enc, $c->param('textInput') ); print $c->header( -charset => 'utf-8' ), $c->start_html("Encoding is + $enc"), $enc, " ", $qtext, $c->end_html; }
      Boris