You haven't told us what database you are talking to and what you are using to load data into the database. Also, we will need to know what the column type is you're inserting into.

When loading the data from Perl, you will need to use Encode to decode the data from its source representation to Unicode.

You will likely need to tell the database driver when loading data into the database and when reading data out of the database that it should consider the column as UTF-8 encoded Unicode (if that's what your original data is).

Alternatively, encode the data to UTF-8 encoded octets and then write those raw octets into the database. You will lose the ability to query the data from within the database in a nice way. LIKE queries and UPPER() will likely not work in the way you expect them.

When reading from the database, set up your database so it decodes the data from your database format to Unicode in Perl.

If you are using MySQL, read DBD::mysql for some UTF-8 options.

I am using MySQl and using the source command with a source file to insert into the table:

insert_test (plaintext source file)
INSERT INTO test VALUES(NULL, "\\x{20ab}");
INSERT INTO test VALUES(NULL, "\\x{005a}\\x{0024}");
INSERT INTO test VALUES(NULL, "\\x{0042}\\x{0073}");

mysql> source C:/Program Files/MySQL/MySQL Server 5.0/sql/insert_test
[download]

Could you enlighten me on a string like this '\x{005a}'? Is it called a unicode string? And is that what we should store in the database (MySQL)?

Hello again Anonymous Monk,

Everything you are looking for is here A UTF8 round trip with MySQL.

More specifically:

use DBI();
    my $dbh = DBI->connect ('dbi:mysql:test_db',
                            $username,
                            $password, 
                            {mysql_enable_utf8 => 1}
    );
[download]

Let the MySQL do all the work for you. By doing this This step connects to the database, and tells DBD::mysql to auto-convert to/from UTF-8..

Hope this helps, BR.

Seeking for Perl wisdom...on the process of learning...not there...yet!

Could you enlighten me on this related issue?

I have a json string from perl (encoded with to_json) which goes to my Javascript code. When I view the json string, it looks like this:

{ "hex_code":"\\x{a5}" }
[download]

I want to display the hex_code as an actual symbol in an input field. But what I see in the input field is '\x{a5}' and not the symbol. In Javascript when I set a variable like this:

var hex_code = '\x{a5}';
document.getElementById('some_field').value = hex_code;
[download]

The symbol gets displayed in the input field.

What do I need to in the perl code (or in the Javascript) to display the hex code as symbol in the input field?

I'm quite confused by this. Hope you can shed some light :)

What do you mean by "view the JSON string"? Do you mean you print it? Do you print it using Data::Dumper?

If you want to turn the JSON back into a proper data structure, I suggest you use JSON or JSON::XS to do that. You might need to unescape the string before doing that if it contains doubled backslashes, so that it is proper JSON:

use JSON 'decode_json';
use Data::Dumper;

my $mangled_json = '{ "hex_code":"\\x{a5}" }';
print $mangled_json;
my $json = $mangled_json;
$json =~ s!\\\\!\\!g;
print $json;

my $structure = decode_json( $json );
$Data::Dumper::Useqq = 1;
print Dumper $structure;

binmode STDOUT, ':encoding(UTF-8)'; # well, hopefully, your terminal u
+nderstands UTF-8
print $structure->{'hex_code'};
[download]