Hello. I am trying to convert "foreign" characters such as é, È, é, etc. into a simple e or E.

The reason is that these complex characters are in our URI's for our site, and we need to canonicalize them into a simple form that can easily be typed (otherwise we would just use numbers or some ID for our URI's). We are not worried about collisions in the converted URI's.

I have looked into such modules as Encode, Unicode::Normalize, and Unicode::Map but I have had very little luck in understanding just even the simple steps I need to take to solve this problem.

If anyone has any advice or even solutions, then I would be very grateful for some knowledge transfer. For what it is worth, the following code is as close as myself and my co-worker have been able to get to solving this problem:

use Encode;
use Unicode::Normalize qw(normalize);

my $ascii = encode('ascii', normalize('KD', $utf8), sub {$_[0] = ''});
[download]

However, even though this worked for him (he used this code on another project/site/client), my output for "é" was "A\x{0303}\x{00a9}" when I was expecting "e" ... perhaps my strings are not UTF-8 after all? Is there a known way to determine what encoding a particular piece of data might be?

Forever puzzled,
Anonymous Monk