Re^4: Encoding of emoji character

Thanks for that. If I use $text = "Test \N{GRINNING FACE}" in my program then the websocket client displays the emoji correctly. I modified your sample code a little and it gave the following output.

With the $text = "Test \N{GRINNING FACE}" for the emoji it gives:

$VAR1 = "\x{1f600}";
128512
GRINNING FACE
[download]

Using the string read from MySQL the emoji gives:

240
LATIN SMALL LETTER ETH
$VAR1 = '&#65533;';
159
APPLICATION PROGRAM COMMAND
$VAR1 = '&#65533;';
152
START OF STRING
$VAR1 = '&#65533;';
128
PADDING CHARACTER
[download]

Changing the database to UTF-8 will be difficult as it's not entirely under our control. Do you think the table being latin1 is the problem?

Comment on Re^4: Encoding of emoji character Select or Download Code

Replies are listed 'Best First'.
Re^5: Encoding of emoji character by soonix (Chancellor) on Jun 21, 2022 at 18:47 UTC
I do think the table being latin1 is a part of the problem. On the other hand, the application that fills the table seems to use a reasonable encoding (UTF-8). If you change the table to something unicodey, that application most probably will NOT automagically insert a unicode character instead of the current 4 bytes. Probably the easier solution will be to check for bytes between 0x80 and 0x9F (because these are not defined for ISO 8859-1, the "official" Latin1). If they are not used otherwise in your variant of Latin1, it might be feasible to try it with Encode::decode. What happens, if you insert something like `{ use Encode qw(decode :fallbacks); $text = decode('UTF-8', $text, FB_WARN); }` [download] after reading $text from the database?	[reply] [d/l]
Re^6: Encoding of emoji character by dcunningham (Sexton) on Jun 22, 2022 at 03:37 UTC
Thank you for the suggestion, but it still died with the "Wide character" error using that. The suggestion further down seems to have resolved it though.	[reply]
Re^5: Encoding of emoji character by Anonymous Monk on Jun 21, 2022 at 06:09 UTC
Interesting. None of these characters have code points above 255, and yet you sometimes get the error in `decode($text)`. You said the table encoding is latin-1. My current guess is, you get your information decoded as if it was latin1. Most of it looks like bytes, but occasionally, latin-1 text decodes to wide characters and blows up `decode` (which only expects bytes). What if you encode `$text` back to latin-1 to get bytes, then decode those as UTF-8? This transformation seems to be reversible as long as all bytes round-trip, that is, MySQL's interpretation of "latin-1" is the same as Perl's and has a meaning for all 256 possible byte values.	[reply] [d/l] [select]
Re^6: Encoding of emoji character by soonix (Chancellor) on Jun 21, 2022 at 18:52 UTC
I think the "wide character" doesn't simply mean "above 255(0xFF)", but "not in the character set". Latin1 defines characters in the ranges 0x00 .. 0x7E and 0xA0 .. 0xFF. Those within 0x7F .. 0x9F are "not within one of the defined ranges", thus "out of range" = "wide character".	[reply]
Re^6: Encoding of emoji character by dcunningham (Sexton) on Jun 22, 2022 at 03:40 UTC
Thank you, this appears to have fixed it! Using the following code the emoji is passed and displayed correctly on the websocket client. `$text = encode( 'iso-8859-1', $text ); $text = decode( 'UTF-8', $text ); $conn->send_utf8( $text );` [download]	[reply] [d/l]