Hi, the process you are talking about is sometimes called "normalization".

We have a catalogue database (of books) parsed from MARC records. When storing things like "title" we also store a normalized value for searching purposes. We follow the NACO normalization rules for our conversion.

Here's the function we currently use. It steals from code from Pod::Escapes to get rid of diacritics... it still needs some work, though (any comments on this function are appreciated).

Note: it doesn't take into account any HTML.

# LUT to convert diacritical and special characters
# Modified from Pod::Escapes
my (%Latin1Code_to_fallback, %Latin1Char_to_fallback);
%Latin1Code_to_fallback = ();
@Latin1Code_to_fallback{0xA0 .. 0xFF} = (
' ', qq{!}, qq{C/}, 'PS', qq{\$?}, qq{Y=}, qq{|}, 'SS', qq{"}, ' ', 'a
+', qq{<<}, qq{!}, "",  ' ', qq{-},
' ', ' ', '2', '3', qq{'}, 'u', 'P', qq{*}, qq{,}, '1', 'o', qq{>>}, q
+q{1/4}, qq{1/2}, qq{3/4}, qq{?},
'A', 'A', 'A', 'A', 'A', 'A', 'AE', 'C', 'E', 'E', 'E', 'E', 'I', 'I',
+ 'I', 'I',
'D', 'N', 'O', 'O', 'O', 'O', 'O', 'x', 'O', 'U', 'U', 'U', 'U', 'U', 
+'Th', 'ss',
'a', 'a', 'a', 'a', 'a', 'a', 'ae', 'c', 'e', 'e', 'e', 'e', 'i', 'i',
+ 'i', 'i',
'd', 'n', 'o', 'o', 'o', 'o', 'o', qq{/}, 'o', 'u', 'u', 'u', 'u', 'y'
+, 'th', 'y',
);

{
  %Latin1Char_to_fallback = ();
  my($k,$v);
  while( ($k,$v) = each %Latin1Code_to_fallback) {
    $Latin1Char_to_fallback{chr $k} = $v;
  }
}

sub normalize {
    my $data = shift;

    # Rules taken from NACO Normalization
    # http://lcweb.loc.gov/catdir/pcc/naco/normrule.html

    # Convert special chars to spaces
    $data =~ s/[\Q!(){}<>-;:.?,\/\\@*%=\$^~\E]/ /g;
    # Delete special chars
    $data =~ s/[\Q'[]|\E]//g;
    # Remove diacritical marks and convert special chars
    my @chars = split(//, $data);
    for (my $i = 0; $i < @chars; $i++) {
        $chars[$i] = $Latin1Char_to_fallback{$chars[$i]} if (ord($char
+s[$i]) >= 160 && ord($chars[$i]) <= 255);
    }
    $data = join('', @chars);
    # Convert lowercase to uppercase.
    $data =~ tr/a-z/A-Z/;
    # Remove leading and trailing spaces
    $data =~ s/^\s+|\s+$//g;
    # Condense multiple spaces
    $data =~ s/\s+/ /g;

    return $data;
}
[download]

--
"To err is human, but to really foul things up you need a computer." --Paul Ehrlich

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.