#!/usr/bin/perl -w
use Tk;
use strict;

my ($mw, $test_string, $txt);
$mw = MainWindow->new(-title => "UTF-8 Test", -width=>400, -height=>30
+0);
$test_string = 'This is a sample Chinese character: 我';
$txt = $mw->Text(-height=>5)->pack;
$txt->insert('end', $test_string);
MainLoop;
[download]

The entity in the code above was inserted by the PerlMonks web site. In my original source code I have a Chinese character. If anyone wants to try out this issue, you'll have to somehow get a Unicode character into the string.
--- Marais

$test_string = 'This is a sample Chinese character: ' . chr(25105);
[download]

use utf8;
[download]

So clearly I have to give this some more thought. It's looking as if a string retrieved from the database must be different in some important way from a constant string. Gack.

Here's another way for that test snippet to work (knowing that 25105. == 0x6211):

#!/usr/bin/perl -w
use Tk;
use strict;

my ($mw, $test_string, $txt);
$mw = MainWindow->new(-title => "UTF-8 Test", -width=>400, -height=>30
+0);
$test_string = "This is a sample Chinese character: \x{6211}";
$txt = $mw->Text(-height=>5)->pack;
$txt->insert('end', $test_string);
MainLoop;
[download]

Works for me, no problem -- I see a Chinese character. In fact, I distinctly remember that whenever I've installed Tk for perl 5.8, the "make test" phase spends a fair bit of time going through all the unicode characters (including the Chinese, Japanese and Korean).

I don't think Tk can do Arabic (or any other right-to-left writing system), and I don't know how well it can do with Devanagari or similar Indic scripts, but all the left-to-right characters work fine.

So if this little test snippet isn't working for you, there might be a problem with your particular installation of the Tk bundle (or maybe it's a font issue?)

If this test snippet works, but you're still having trouble getting stuff from mysql to display correctly (even though you seem to be decoding it as you should, or have "utf8_enabled" in the DBI connection), then you may want to check other modes of access to the DB, to see if the actual content in your tables is not what you expect.

did you try  mysql_enable_utf8?

Thanks for the suggestion. I gave it a try, and got the following error message:

Tk::Error: Cannot decode string with wide characters at /usr/lib/perl/5.8/Encode.pm line 166.

A UTF-8 character pasted into one of the Text widgets does end up in the database correctly, so my feeling is that communications between the app and the database are not the problem.

Here's my test program:

#!/usr/bin/perl -w
use strict;
use utf8;

my (
 $codepoint,
 $test_string,
 );

 binmode STDOUT, ":utf8";
 
 $codepoint = ord('我');
 print "Codepoint of character is $codepoint\n";

 $test_string = "Here's a test string with 我\n";
 print "test_string is $test_string\n";
 
 $test_string_dec = decode('utf8', $test_string);
[download]

There are at least two issues here:

1. The presence of the "use utf8" pragma: I gather from Googling that this is no longer required in current versions of Perl (I'm using 5.8.8.) But, if I leave it out, the codepoint reported by "ord" is 250, instead of 25105. Surely Perl should know that the Chinese character is Unicode?
2. The "binmode" statement: This interacts with the "use utf8" pragma in the following ways:
   both present: correct codepoint, correct output character, no error message
   pragma present, binmode omitted: correct codepoint, correct output character, "Wide character in print" error message.
   pragma omitted, binmode present: wrong codepoint, wrong output character, no error message
   Both omitted: wrong codepoint, output shows correct Chinese char, no error message.

So, I'm really confused. What does the "use utf8" pragma actually do in Perl 5.8.8? Why do I get the correct character showing on output even when I get the "Wide character in output..." message?

--- Marais