The problem is that the legacy set makes use of the extended ascii character set (8-bit chars) which don't convert to Unicode (easily).
Hmm... why not?
My take when asked about it was: don't! Keep two lists for lookup and don't mix them, because they cannot logically be sorted together. They countered by sorting two small subsets together (using Java) and saying that it was easier for their people to do lookups in a single list.
Well, that doesn't look too difficult? Why not decode their legacy set (as in
map Encode::decode( 'LEGACY_SET', $_ ), @set) and sort that? And if their set happens to be ISO-8859-1 (aka Latin-1), then decoding isn't even necessary (and that's the deal with
utf8-off strings in perl; they're assumed to be in THAT encoding, although
some people say it just looks like it :)
The result of this thread is so depressing that I'm going to turn the work down and let them find someone else. (Shame. Could have been a nice in.)
Shame indeed, because Perl is actually very good for Unicode stuff...
Unicode::Collate::Locale, for example... but yeah, Perl's strings are a source of much confusion.