in reply to Comparing inputs before inserting into MySQL for "sounds-like" similarity
Decouple the article import from detecting duplicates - the latter process could be quite involved, and it will allow you to play with the detection algorithm without interrupting your import process.
Have a look at this article: Building a Vector Space Search Engine in Perl.
What you need to do could be seen as an instance of the document classification problem. A search engine would help find documents would could be duplicates of a target document, and then you could use something like Text::Similarity to perform a more robust comparison (or flag them for manual review.)
|
|---|