in reply to Encoding/decoding for korean strings

Sounds like you either need to give the EV stuff a UTF-8 layered file handle (which may not be an option; see UTF-8 caveats in syswrite) or you need to encode_utf8() the characters you're giving to it back to bytes for it to write. I poked around the sources for Mojo::Reactor::EV and IO::Handle and did not see a clear place to suggest more. If you can give a minimal example that exhibits the problem, I guarantee someone here will be able to help. The mojo IRC would probably be a place to get immediate help too. Please post your solution here if you arrive at it elsewhere. :P

Replies are listed 'Best First'.
Re^2: Encoding/decoding for korean strings
by siva_pm (Novice) on Jun 15, 2016 at 19:11 UTC
    I tried to use the encode_utf8() and it doesn't complaint in that . However, this corrupts the characters on the UI and they're not korean anymore and when I try to decode_utf8(encode_utf8($mystring)) it complaints that it can't write wide character again.

      Every single layer has to agree upon what it's looking at from the input (JSON is always UTF-8) to the parsing to the writing to the DB to the fetching from the DB to the UI. So if the characters are being corrupted, it might be because your DB is actually expecting EUC-KR or something. In that case you might try encode("EUC-KR", $utf8_string); check your DB schema and see what it expects. But the problem might be in the UI uptake or output or your DB might be incorrectly configured/specified.

      Update: didn't see your reply before I replied. I might look at this later but it's a lot to digest. :(

        So I have my db accepting these korean characters and can see them in korean in the tables too. I cannot use EUC-KR as I might have to accept all kinds of languages as input (one byte/two byte)

      Below are the methods that deal with the encoding/decoding So I'm calling handleRequest($fun,$parameters) from my cgi file which calls a Handler.pm which initially does an eval call to handleRequest which firstly does eval call looks like this:

      sub handle_req_cgi{ eval { $log->info("pjb.cgi - Started function $function , user: $user_ +name"); $msg = UI::PJB::handleRequest($function, $parameters); 1; } //Response $tx->res->code($res_code); $tx->res->headers->content_type('application/json'); $tx->res->body(encode('UTF-8',$msg)); #THIS IS THE LINE which corrupts the msg but if I print just the msg w +ithout encode then #wide character warning is thrown by Mojo $tx->rendered; }
      handleRequest() # looks like this: { $jsonDecodedParameters = $J->decode($parameters); my $result = executeFunction($fun, $jsonDecodedParameters); if(ref $result eq ref {} && defined $result->{$HTTP_REQUEST_REDIRECT_U +RL_KEY}) { $jsonEncodedResult = $J->encode($result); returns this $jsonEncodedResult }

      then the control inside Handler.pm proceeds to write the response of

        Maybe there's something I'm not understanding, but there's nothing in the first snippet that calls the "handle_pjb_cgi" function in the second snippet, and so I have no idea what the "$tx" thing is. Assuming "handle_pjb_cgi" is being passed the sort of object it should be getting, the description in the OP makes it sound like whatever file handle is being used to "print" the the "message" has not been set to use a utf8 layer.

        I gather from your other reply below that you've confirmed there is valid utf8 data stored in the database, so the problem would seem to be limited to the output process, and probably involves the discipline setting on the output file handle. But I'm just guessing here.