in reply to Re: Fuzzy text matching... again
in thread Fuzzy text matching... again

  • Remove unimportant tokens (trailing text in parens)

You certainly have a number of good points.  However, parenthesization need not necessarily imply subordination, it could also stand for an alternative name/term, as in the OP's case #5:

where the part in parentheses has a better chance of contributing to a successful match than the unparenthesized part.  Just to illustrate one of the many potential issues the OP might encounter.

And while we're at it: how would a machine identify what is a name and what not - as in "Archivio Giuliano Marini" - without consulting either a database of common names, or checking against a list of all known regular words (+ inflections) in a particular language?   Even Google translate apparently gets it wrong when translating "Archivio Giuliano Marini" into English (leaving "Archivio" as is, instead of translating it to "archive" — even though you tell it what source language it is), while it gets it right (interestingly) with "Archivio Marini"...

Replies are listed 'Best First'.
Re^3: Fuzzy text matching... again
by Limbic~Region (Chancellor) on Jan 07, 2010 at 16:27 UTC
    almut,
    The determination of what is unimportant is something the OP will have to come up with. To be honest, until you pointed it out as a translation, I had no idea. Obviously nothing will be perfect, which is why I like the "prospector" method and continous refinement. I know it isn't bayesian filter heuristics but I have been quite successful with it.

    And while we're at it: how would a machine identify what is a name and what not

    I can't remember mentioning names anywhere in my response. Name matching has a completely different kind of complexity (Mark and Mary only have an edit distance of 1; Richard, Rich, Dick, Dickie may all refer to the same person; cultural, ethnic and religious variations to a name; gender; variations in converting from native to latin-1; etc). And yes, there are huge databases of names (common and otherwise) to deal with this. Check out Global Name Recognition software owned by IBM for a very expensive solution to that problem.

    Of course, names are not the only specific kinds of strings that have their own complexity. Mailing addresses, dates, identifying an anonymous author, identifying plagerism, indexing, etc. I limited my advice to the problem as I understood it. I obviously could have missed the mark thinking these were publication citations making the "what is a name" germane but I don't see why it can't just be treated like any other token. I obviously missed the trailing parens sometimes being important as a translation but I assume the OP will be capable of constructing an approach for removing unimportant tokens.

    Cheers - L~R

[reply]
      I can't remember mentioning names anywhere in my response.

      My last paragraph wasn't meant as a reply to anything you've said in particular, more as a "P.S." — Sorry for not having made it clear.

      ...these were publication citations making the "what is a name" germane but I don't see why it can't just be treated like any other token.

      I just wanted to point out that if you understand what is what in "Archivio Giuliano Marini" you have a much better chance of telling if something else like "Archivio Giuliano Cassini" is referring to the same thing or not. I.e., knowing that "Archivio" simply means "archive" and that "Giuliano" is a common given name, you'd most likely figure out that they're two different archives, while a simple token comparison might identify them as being the same (due to two of three tokens matching, and the third one sounding similar).

[reply]

In Section Seekers of Perl Wisdom