shemp has asked for the wisdom of the Perl Monks concerning the following question:

I am working on an app that tries to parse text that may be very poorly formatted. The text is mainly public record data, and was mostly entered into various systems many years ago, by people who never thought about algorithmic parsing. They jsut entered the text into the field with all sorts of goofy abbreviations.
My algorithm is trying to clean up some of the inconsistencies, for instance the word "Trust" may be abbreviated as "Trst", "Tst", "Tru", etc.
So, im thinking of using String::Approx to match potential abbreviated words against a pre-determined list. Would this be wise, or does anyone know of a module more specific to my needs?

thanks

Replies are listed 'Best First'.
Re: abbreviation checking
by gjb (Vicar) on Dec 02, 2002 at 19:05 UTC

    Frankly, I don't think that Text::Soundex is the way to go, it is most probably too coarse a measure for what you want to achieve.

    Apart from String::Approx there's also Text::WagnerFischer and Text::Levenshtein, Both are string distance measures. I'd use a gradual refinement, starting with a large threshold value to collect candidates to be replaced and taking smaller values until the noise level is acceptable. If the lists of words are not too large I'd valiidate it manually so that each word corresponds to the right set of abbreviations. After that, it's a simple matter of substituting strings.

    Just my 2 cents, -gjb-

    Update: You might also want to have a look at Text::KeyboardDistance to catch typos.

[reply]
Re: abbreviation checking
by dree (Monsignor) on Dec 02, 2002 at 20:34 UTC
[reply]
      While making an MP3-renaming script, which attacks a problem similar to yours, I used a combination of Metaphone and "distance" modules. My approach:
      1. Get a list of "known-good" words. I use already-verified MP3 filenames as a source of these.
      2. Calculate their Metaphones.
      3. Calculate the Metaphone of any new words and look for matches. If none, see if there are any matches with a distance of 1 or 2. Distances larger than 2 produce too many matches.
      4. Have the user confirm the 'corrections'.
      It's not an exact science, and human intervention is unavoidable if correctness matters.
[reply]
Re: abbreviation checking
by BrowserUk (Patriarch) on Dec 02, 2002 at 18:16 UTC

    Text::Soundex might help you.

    Okay you lot, get your wings on the left, halos on the right. It's one size fits all, and "No!", you can't have a different color.
    Pick up your cloud down the end and "Yes" if you get allocated a grey one they are a bit damp under foot, but someone has to get them.
    Get used to the wings fast cos its an 8 hour day...unless the Govenor calls for a cyclone or hurricane, in which case 16 hour shifts are mandatory.
    Just be grateful that you arrived just as the tornado season finished. Them buggers are real work.

[reply]
Re: abbreviation checking
by Anonymous Monk on Dec 03, 2002 at 00:41 UTC
    I have a similuar problem expect it deals with names and address.
    You know 
    John Smith
123 North East First St.
Doofus Ville, GA 31314

    [download]
    Should match 
    J. Smith
123 NE 1 St.
Doofus Ville, Geogria 31314

    [download]
    Anybody got any good packages for that?
[reply]
[d/l]
[select]
[reply]

Back to Seekers of Perl Wisdom