First, let's clear some confusion. Unicode doesn't specify how characters are stored, so you can't possible be talking about Unicode when you're talking about a string of bytes. It looks like you meant UTF-8 when you said Unicode. UTF-8 is a means of representing (encoding) Unicode characters in bytes.

$string =~ s/([^a-zA-Z0-9])/'&#'.unpack('U0U*',$1).';'/eg;
[download]

can also be written as

use HTML::Entites qw( encode_entities );
$string = encode_entities($string);
[download]

and

use Encode qw( encode );
$string = encode('US-ASCII', $string, Encode::FB_HTMLCREF);
[download]

No need to reinvent the wheel.

If you use the latter, you can combine the decoding and encoding into one step.

use Encode qw( from_to );

sub unicode_decode
{
  my $string = shift;
  from_to($string, 'UTF-8', 'US-ASCII', Encode::FB_HTMLCREF);
  return($string);
}
[download]

Thanks to both of you.
The thing is, $string must not contain certain characters like comma and slash, since I use those as separators in my text files. Thats what the regex also ensures, so I think its still the best choice here given the circumstances.
So I now go with


use Encode qw(decode);

sub unicode_decode
{
  my $string = decode('utf8', shift, 0);
  $string =~ tr/\x{FFFD}/\x20/;
  $string =~ s/([^a-zA-Z0-9\_\+\-\.])/'&#'.unpack('U0U*',$1).';'/eg;
  return($string);
}
[download]

As you can see, this also swaps the replacement character with a space should there be one.

I think your method (in your later reply) of using a space to replace each "\x{FFFD}" (the unicode replacement character, which is inserted whenever there is an "uninterpretable" byte sequence) is as good as any, though maybe the "ellipsis" character ("\x{2026}" or "\x{22ef}") would be more appropriate.

IMHO, anyone who goes to the trouble of creating a "name" that contains both Latin-based (left-to-right) and Arabic-based (right-to-left) characters in a single word token is most likely trying to make trouble, and should expect (presumably wants) to see things go wrong.

You're right, it is a limit, presumably for the names being stored in a 32 byte string with null-termination.

But you can't possibly suggest those people to be looking for trouble, as most are merely kids trying to appear "cool" with a fancy name, and many don't even know what Unicode or UTF-8 is. They simply gather some nice looking characters from a character map and assemble a name as if they were playing with LEGO bricks.

For anyone who wants to see the script in action, you can find it on http://forlix.org/ (the table on the bottom right). I have also added some whitespace treatments, so multiple spaces wont be collapsed (the CSS solution white-space:pre isn't yet supported well enough)

my $n = "\x78\x54\xC5\x99\xD8\xB9\xD0\xBC\xD8\xB9\x2E\x20\xE1\xB8\xA0\
+x7C\x20\x7
B\xC4\xA2\x6C\xC3\xA2\xC3\x90\xC3\xAE\xC3\xA2\x54\xC5";
my $m = $n; chop $m;

say 'Decode error with \xC5' unless utf8::decode ($n);
say 'Decode error without \xC5' unless utf8::decode ($m);

say "With \\xC5: $n";
say "Without \\xC5: $m";
__END__
[download]

Note that with the trailing \xC5, utf8::decode detects an error.