in reply to Re^2: Fuzzy text matching... again
in thread Fuzzy text matching... again

almut,
The determination of what is unimportant is something the OP will have to come up with. To be honest, until you pointed it out as a translation, I had no idea. Obviously nothing will be perfect, which is why I like the "prospector" method and continous refinement. I know it isn't bayesian filter heuristics but I have been quite successful with it.

And while we're at it: how would a machine identify what is a name and what not

I can't remember mentioning names anywhere in my response. Name matching has a completely different kind of complexity (Mark and Mary only have an edit distance of 1; Richard, Rich, Dick, Dickie may all refer to the same person; cultural, ethnic and religious variations to a name; gender; variations in converting from native to latin-1; etc). And yes, there are huge databases of names (common and otherwise) to deal with this. Check out Global Name Recognition software owned by IBM for a very expensive solution to that problem.

Of course, names are not the only specific kinds of strings that have their own complexity. Mailing addresses, dates, identifying an anonymous author, identifying plagerism, indexing, etc. I limited my advice to the problem as I understood it. I obviously could have missed the mark thinking these were publication citations making the "what is a name" germane but I don't see why it can't just be treated like any other token. I obviously missed the trailing parens sometimes being important as a translation but I assume the OP will be capable of constructing an approach for removing unimportant tokens.

Cheers - L~R

Replies are listed 'Best First'.
Re^4: Fuzzy text matching... again
by almut (Canon) on Jan 07, 2010 at 18:03 UTC
    I can't remember mentioning names anywhere in my response.

    My last paragraph wasn't meant as a reply to anything you've said in particular, more as a "P.S." — Sorry for not having made it clear.

    ...these were publication citations making the "what is a name" germane but I don't see why it can't just be treated like any other token.

    I just wanted to point out that if you understand what is what in "Archivio Giuliano Marini" you have a much better chance of telling if something else like "Archivio Giuliano Cassini" is referring to the same thing or not. I.e., knowing that "Archivio" simply means "archive" and that "Giuliano" is a common given name, you'd most likely figure out that they're two different archives, while a simple token comparison might identify them as being the same (due to two of three tokens matching, and the third one sounding similar).