Jim has asked for the wisdom of the Perl Monks concerning the following question:

Fellow Monks,

I'm accessing text in a Microsoft Access database via DBI and DBD::ODBC that's stored in the Windows-1252 character encoding. I've proved to myself that it's coming out as Windows-1252 and not Latin 1 (ISO-88591-1) or UTF-8. I want to convert the characters to UTF-8 and then to Java Unicode Notation. For example, I want the Euro sign in Windows-1252 (\x80) to be printed to a text file as its equivalent Unicode character U+20AC transformed into the string '\u20ac'. Is there something ready-made for this purpose? I know about the Encode module, but I'm looking for the canonical solution to the last bit: converting the resultant UTF-8 into Java Unicode Notation.

I've read too much documentation and now I'm confused. I want the simplest, most elegant solution.

Thanks!

Jim

  • Comment on Convert Windows-1252 Characters to Java Unicode Notation

Replies are listed 'Best First'.
Re: Convert Windows-1252 Characters to Java Unicode Notation
by Juerd (Abbot) on Nov 25, 2007 at 21:48 UTC 

    my $uni = decode("Windows-1252", $input);
$uni =~ s/(.)/sprintf "\\u%04X", ord $1/ge;
print $uni, "\n";

    [download]

    If you only want to change certain characters, change the . part to match only what you want to change to \u encoding.

    Juerd # { site => 'juerd.nl', do_not_use => 'spamtrap', perl6_server => 'feather' }

[reply]
[d/l]
      Thank you very much, Juerd. This worked brilliantly:

      # Convert Windows-1252 characters into Java's Unicode notation...
$md->{$column} =~ s{([\x80-\xFF])}{
    sprintf "\\u%04x", ord decode('cp1252', $1)
}eg;

      [download]
      Simple and elegant.

      (By the way, the frequency of occurrence of non-US-ASCII characters in the data is very low in relation to the amount of text. So-called 8-bit characters are infrequent and usually occur in isolation.)

      Jim

[reply]
[d/l]

        You may be better off with a hard coded translation table, for performance. 

        my %w1252_to_java = map {
    chr($_) => sprintf("\\u%04x", decode "Windows-1252", chr)
} 0x80 .. 0xff;

...

$md->{$column} =~ s/([\x80-\xff])/$w1252_to_java{$1}/g;

        [download]

[reply]
[d/l]
        (By the way, the frequency of occurrence of non-US-ASCII characters in the data is very low in relation to the amount of text. So-called 8-bit characters are infrequent and usually occur in isolation.)
        Maybe in English, but not when your data is in French, for example. In French you can easily have one or two accented characters every other word.

        Ain't it typical again that English speaking people automatically assume that the whole world uses only English...

        Well, I'm assuming that now you're just talking about your own, personal case. Yes, in that case it's very likely that accented characters are very rare. Until you start getting an international audience, that is...

        BTW the difference between ISO-Latin-1 and Windows-1252 will most probably be most visible in the so-called "smart quotes", those curly quotes that bend a different way for opening and closing quotes, and the ditto curly apostrophe.

[reply]

Back to Seekers of Perl Wisdom