i've been toying with unicode lately working on a project that involves transliterations of tibetan and sanskrit texts. they use all sorts of weird characters like Ā (an A-macron (ie, an A with a bar over it)) which has the numerical value of 256 and is written as Ā for html/xml but is internally represented by two bytes 0xC4 0x80 (196 128).

since this has to go on the web, be processed by some XML parsers, and be stored in a database, the best solution i could find for storing the chars is as numerical XML entities (Ā). my problem now is finding a way to reliably convert the unicode to these numerical entities. if you use the standard HTML::Entities module like so:

use HTML::Entities;
my $text = chr(256);
print encode_entities($text);

[download]

you get 'Ä€' because it isn't smart enough to look at it as one unicode character instead of two ascii characters.

a little digging and i found the Text::Iconv module. tried it out:

use HTML::Entities;
use Text::Iconv;
my $text = chr(256);
my $conv = Text::Iconv->new("utf8", "latin1");
$text = $conv->convert($text);
encode_entities($text);
print "$text\n";

[download]

this doesn't seem to work either. (i get an 'uninitialized value in substition' warning from HTML::Entities and no output. converting it to latin1 isn't going to work in the long run anyway because it's way too lossy; i need to retain all kinds of characters that aren't going to convert to latin1 at all like t-underdot (ṭ (ṭ if you have the right unicode fonts installed)).

not being able to find any more promising looking modules, i broke out the pen and paper and figured out some math, which gave me the following code:

$text =~ s/([^[:print:]])([^[:print:]])/
          '&#'.
          ((((ord($1) & 0x7F) << 6) + (ord($2) & 0x7F)) & 0x7FFF).
          ';'/eg;

[download]

the math's a bit much to explain here. basically it looks for a sequence of two POSIX non-printable bytes and combines them into a single decimal number sandwiched between '&#' and ';'. it works ok for chars between 256 and 2047 but doesn't seem to cover the whole range. furthermore, i feel like the approach is a dead end since it would make mistakes when it saw two perfectly legal chars in a row like 'éá' thinking that the two of them together were a unicode character converting them to a single entity. typical problem with a roll-your-own solution.

so what i'm looking for is essentially HTML::Entities but with the ability to correctly handle multi-byte unicode characters. does such a beast exist?

anders pearson

In reply to converting unicode to numerical entities by thraxil

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.