Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW

Re: How do I normalize (e.g. strip) diacritical märks from a Unicode string?

by brycen (Monk)
on Apr 17, 2010 at 07:36 UTC ( #835240=note: print w/replies, xml ) Need Help??

in reply to How do I normalize (e.g. strip) diacritical märks from a Unicode string?

Unicode defines a variety of normalization forms (see

I prefer normalization form NFKD, as it translates more ligatures (though not all, for example the ligature Œ).

First decompose composite characters into their component parts (e.g. letters and diacritical marks), then strip out the marks.

$str = Unicode::Normalize::NFKD($str); $str =~ s/\p{NonspacingMark}//g;
Or with a full example:
## Demonstrate stripping of diacritical marks from Unicode strings ## April 2010, Bryce Nesbitt, Berkeley Electronic Press ## See also ## See also ## Keywords: perl, diacritic, diacritical ## accent, iso-8859-1, normalization. use utf8; # Tell perl source code is utf-8 use 5.10.0; use Unicode::Normalize; # Sample: "latin small letter e with circumflex and tilde" &#7877; # "latin small ligature ff" (will be expanded) # "latin small ligature oe" (won't be expanded) $str = shift || "\x{1ec5} märks \x{fb00} \x{153}"; say "Input: ".debug_chatty_string($str); # Decompose into letter and combining marks, in "Kompatibility" mode $str = NFKD($str); say "NFKD : ".debug_chatty_string($str); # Remove combining marks $str =~ s/\p{NonspacingMark}//g; $str = lc($str); say "Out : ".debug_chatty_string($str); sub debug_chatty_string { my $outstring; # Use shift below, so utf-8 flag is preserved. # Else you might have to fiddle with Encode::_utf8_on() foreach $char (split //,shift) { my $ord = ord($char); if(($ord >= 32 && $ord < 127) || $ord == 10) { $outstring .= $char; } else { $outstring .= "<0x".sprintf("%x",$ord).">"; } } return $outstring; }
Example run:
Input: <0x1ec5> m<0xe4>rks <0xfb00> <0x153> NFKD : e<0x302><0x303> ma<0x308>rks ff <0x153> Out : e marks ff <0x153>

Update: I really do mean normalization. ASCIIfying (e.g. encoding) would destroy non-latin text. Normalization preserves Greek, Hebrew, etc.

I am supporting clients in various languages who want the fuzzy matching that stripping diacriticals provides. It might make for the occasional confusion between German bears and bars... but that's much better than missing out on all the potential correct matches. For example in Hebrew vowels are not normally written except for children. Stripping the vowel and pronunciation diacriticals out lets you compare the text as an adult searcher will likely enter it.

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://835240]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others examining the Monastery: (3)
As of 2022-12-09 00:06 GMT
Find Nodes?
    Voting Booth?

    No recent polls found