Make some independant patterns:
Once you have these make a loop that will match all the patterns on each entry, where pattern groups like 'city_state' are per match...my %regexes = ( zip => { 100 => qr/(\d{4,5})/, 20 => qr/(\d+)/ }, city_state => { 200 => qr/(\w+)\s+([A-Z]{2})/, 10 => qr/(\w+)\s+(( +?:\w+)+)/), city => { 25 => qr/(\w+)/, 2 => qr/((?:\w+)+)/ }, name => { 50 => qr/^\s+(\w+)/ }, ... );
The numbers keying the regexes are scores to assign for each match.
Have the computer display all possible interpretations of the address in a very well formed manner (print "Zipcode: <<$zip_match>>";), sorted by the match score, and let the user choose which one to use.
As you go through some records you can tweak your scores a little, and maybe introduce a threshold parameter (automatically select the first match if it's score is 10 times more than the next).
This should let you avoid cutting and pasting, or other tedious mouse/kebyoard work which is probably going to be the bulk of your time. If you can get 50% of the addresses handled automatically I guess you could weed through even 10,000-20,000 in less than a few hours of work - just add some checkpointing mechanism to the script so that you can go back and redo a single entry, or stop the script, delete bad parses from the database, and start the script again, and it will continue to parse the next unparsed entry.
Good luck!
Update: some guages to give a score - use less variadic types, try to anchor the matches, and align them on field numbers... The more constraints you put on a pattern, the higher it's score should be, but do keep around low scoring fallback matches. Try to match patterns in groups, like city_state, street_city, street_city_state, and give the latter one more than the sum of the other two, so that it will be preferred to fill those fields.
Also - don't be too generous with the number of alternatives - you have to permutate N arrays of matches to produce N*N possible parses, and even if you clip the lowes scoring matches, it might be overwhelming.
If you know the list of cities in advance, programmatically create a regex to match them as alternatives. Do this for states regardless - if there is a two letter word and it's not a statename, it better not be parsed as such. Do this for long words too, e.g. "New York" (even though that one is ambiguous).
Also make a "Postpone" choice, that just appends the entry to the file, and moves to the next one (possibly making a stub entry in the database), so that you can delay anything that isn't obvious at once till later, and deal with it when you're not in auto-pilot mode.
In reply to Re: Data Salad Address Problem
by nothingmuch
in thread Data Salad Address Problem
by SamCG
| For: | Use: | ||
| & | & | ||
| < | < | ||
| > | > | ||
| [ | [ | ||
| ] | ] |