Comparing likeness of two strings

RiotTown has asked for the wisdom of the Perl Monks concerning the following question:

I'm in the process of putting some anti-fraud tools into place for my company. Part of this is to compare strings from an input form against those already existing within a database (all free form). IE, comparing "Steven Smith" to "Stephen Smithe" and "Steve Smith". This isn't limited to names right now, just makes for easier explaining.

I've got an algorithm and query already set up to return rows that are close to the user-input text, but I'm running into problems displaying exactly how similar the two strings are for ranking purposes.

Currently I'm testing with good results the Levenshtein Distance algorithm. The problem is that this method is SLOW, especially when comparing longer strings. (info can be found here on the Levenshtein Distance algorithm.

Does anyone know of a faster way to get a meaningful representation of how closely two stings match?

Comment on Comparing likeness of two strings

Replies are listed 'Best First'.
Re: Comparing likeness of two strings by waswas-fng (Curate) on Jan 19, 2005 at 20:29 UTC
Text::LevenshteinXS looks like it may be faster than the pure perl implementation. Also you can get other viewpoints with added info from modules like: Text::WagnerFischer and one of the soundex or metaphone modules on cpan. -Waswas	[reply]
Re: Comparing likeness of two strings by VSarkiss (Monsignor) on Jan 19, 2005 at 20:24 UTC
The String::Approx module uses Levenshtein distance for its matching. According to the docs, the latest version is "2-3 times faster" than the previous one. Give it a try. Do not rebuke them with harsh words ... but rather lead them gently - with URLs - so that they may learn wisdom.	[reply]
Re: Comparing likeness of two strings by Limbic~Region (Chancellor) on Jan 19, 2005 at 20:27 UTC
RiotTown, It sounds like you rolled your own. Text::Levenshtein has a fastdistance function and if that is not fast enough for you, Text::LevenshteinXS uses C for speed. Cheers - L~R	[reply]
Re: Comparing likeness of two strings by RiotTown (Scribe) on Jan 19, 2005 at 22:04 UTC
I'm going to give the Text::LevenshteinXS module a try to find out if that implementation is really significantly faster than the pure perl function that I've got now. Thanks for the input!	[reply]
Re^2: Comparing likeness of two strings by RiotTown (Scribe) on Jan 19, 2005 at 22:15 UTC
Huge improvement. I've seen increases in performance of 2-3x over the pure Perl. Once again, thanks!	[reply]