Re^3: Normalizing diacritics in (regex) search

Sorry for the delay in response, but there have been distractions, I had trouble finding the code, and then dithered over whether to just put it somewhere public and point to it (I decided not to).

Assuming $in has been properly decoded, I was proposing something like the following:

use Unicode::Normalize qw{ NFKD };
...
my $out = NFKD( $in );
$out =~ s/ \p{NonspacingMark}+ //smxg;

Note that this does not handle anything but diacritics. The above will change 'Köln' to 'Koln', but 'Øslo' (if it were really spelled that way) remains 'Øslo', because Unicode does not consider the stroke to be a diacritic.

I think that for comparing things Unicode::Collate is actually the way to go

Comment on Re^3: Normalizing diacritics in (regex) search Download Code

Replies are listed 'Best First'.
Re^4: Normalizing diacritics in (regex) search by LanX (Saint) on Nov 26, 2025 at 21:02 UTC
Thanks, very similar to jo37's solution (and not really using Unicode::Collate like you suggested° ;-) But jo37's approach with NFD is IMHO better because of the "dangers of pathological characters" I mentioned... Consider U+3374 ㍴: NFKD will decompose it to "bar", NFD won't. That means a symbol/character "㍴" might match in "Barbra Streisand". So if eliminating diacritics is the goal, NFD is preferable. °) For completeness: There is an example in Unicode::Collate, demonstrating normalized search with a (broken°) German phrase e.g. when the content of $str is "Ich muß studieren Perl.", you say the following where $sub is "MüSS", `my $Collator = Unicode::Collate->new( normalization => undef, level => + 1 ); # (normalization => undef) is REQUI +RED. my $match; if (my($pos,$len) = $Collator->index($str, $sub)) { $match = substr($str, $pos, $len); }` [download] and get "muß" in $match, since "muß" is primary equal to "MüSS". Alas I didn't "study" this module sufficiently to tell if this is exactly matching my requirements to only ignore diacritics. Cheers Rolf _{(addicted to the Perl Programming Language :) see Wikisyntax for the Monastery} °) Ha :) ... you can almost hear an English accent with this word order, OTOH I suppose it's easier to decipher for English speakers than "Ich muss Perl studieren". (Which is still slightly off, "lernen" would be better in this case)	[reply] [d/l]
Re^4: Normalizing diacritics in (regex) search by LanX (Saint) on Nov 27, 2025 at 13:16 UTC
> Note that this does not handle anything but diacritics. The above will change `Köln` to `Koln', but 'Øslo` (if it were really spelled that way) remains 'Øslo', because Unicode does not consider the stroke to be a diacritic. weirdly enough, there are Combining Diacritical Marks listed for strokes ̷ `U+0337 ̷ 823 Combining Short Solidus Overlay` ̸ `U+0338 ̸ 824 Combining Long Solidus Overlay` but the effects are not the same ŵôr̂d̂ w̷o̷r̷d̷ w̸o̸r̸d̸ While listed as diacritics they seem only to be used for `<strike>` like negation. > but `Øslo` (if it were really spelled that way) It isn't, but you can take smørrebrød° for the `LATIN CAPITAL LETTER O WITH STROKE` :-) Cheers Rolf _{(addicted to the Perl Programming Language :) see Wikisyntax for the Monastery} °) literally a smeared-bread	[reply] [d/l] [select]