i've been toying with unicode lately working on a project that involves transliterations of tibetan and sanskrit texts. they use all sorts of weird characters like Ā (an A-macron (ie, an A with a bar over it)) which has the numerical value of 256 and is written as Ā for html/xml but is internally represented by two bytes 0xC4 0x80 (196 128).

since this has to go on the web, be processed by some XML parsers, and be stored in a database, the best solution i could find for storing the chars is as numerical XML entities (Ā). my problem now is finding a way to reliably convert the unicode to these numerical entities. if you use the standard HTML::Entities module like so:

use HTML::Entities;
my $text = chr(256);
print encode_entities($text);
[download]

you get 'Ä€' because it isn't smart enough to look at it as one unicode character instead of two ascii characters.

a little digging and i found the Text::Iconv module. tried it out:

use HTML::Entities;
use Text::Iconv;
my $text = chr(256);
my $conv = Text::Iconv->new("utf8", "latin1");
$text = $conv->convert($text);
encode_entities($text);
print "$text\n";
[download]

this doesn't seem to work either. (i get an 'uninitialized value in substition' warning from HTML::Entities and no output. converting it to latin1 isn't going to work in the long run anyway because it's way too lossy; i need to retain all kinds of characters that aren't going to convert to latin1 at all like t-underdot (ṭ (ṭ if you have the right unicode fonts installed)).

not being able to find any more promising looking modules, i broke out the pen and paper and figured out some math, which gave me the following code:

$text =~ s/([^[:print:]])([^[:print:]])/
          '&#'.
          ((((ord($1) & 0x7F) << 6) + (ord($2) & 0x7F)) & 0x7FFF).
          ';'/eg;
[download]

the math's a bit much to explain here. basically it looks for a sequence of two POSIX non-printable bytes and combines them into a single decimal number sandwiched between '&#' and ';'. it works ok for chars between 256 and 2047 but doesn't seem to cover the whole range. furthermore, i feel like the approach is a dead end since it would make mistakes when it saw two perfectly legal chars in a row like 'éá' thinking that the two of them together were a unicode character converting them to a single entity. typical problem with a roll-your-own solution.

so what i'm looking for is essentially HTML::Entities but with the ability to correctly handle multi-byte unicode characters. does such a beast exist?

anders pearson