eyepopslikeamosquito,
What I ended up doing was breaking up the addresses into "buckets" by postcode and comparing all addresses with all other addresses in the same bucket only. To ensure reasonable reliability of postcodes, I had another program to compare suburb names with postcodes, looking for errors.
Let's say my input address is:
123 Broadway Av
Brownville, ME 04415
Let's say the correct best DB address to match is
123 Brodway Avenue
Brownville Junction, ME 04451
How would your approach work?
| [reply] |
It's a long time ago, but I'll try to remember how this address de-duplication program operated.
The addresses were first parsed into an "internal (canonical) form": for example, "Av" and "Avenue" above would parse to the same internal number. The "internal forms" were compared, not the original text. Addresses that were too weird and couldn't be parsed were rejected for inspection by a human. As part of the parsing, suburbs were checked against an Australian postcode database and errors reported, so the postcodes were cleaned (sometimes by a human). Suburbs were then ignored, postcodes were considered more reliable (people sometimes lie about their suburb name in Australia because some suburbs are "posher" than others).
There was also a user-configurable "number of transcription errors" allowed; if you allowed one transcription error, then "Broadway" and "Brodway" above would be considered a match. There was another parameter for transposition errors (e.g. "Gergory" versus "Gregory") and many other heuristics, like "Flat 6" versus "Flat 6A" versus "Apartment 6".
Update: Perhaps a better technique, which I didn't know about at the time and as suggested by metaperl, is Jaro-Winkler distance. I remember trying classical Knuth Soundex and found it to be not useful in this domain.
One of the many imperfect and sometimes amusing heuristics I remember was "Captain Cook Cruises": is this a gentleman with a title of Captain, first name Cook, last name Cruises, or is it a company? Sydneysiders will know the likely answer :-). How about "Mr William and Mrs Melinda Gates": is this "Mr William" and "Mrs Melinda Gates" or "Mr William Gates" and "Mrs Melinda Gates"?
Update: No matter what (imperfect) heuristics you use for the address parsing and matching, you still face the infeasibility of comparing each address with every other one, as n becomes large. An obvious way to tackle that combinatorial explosion is to partition the addresses into "buckets" and compare only addresses within the same bucket. While various "bucketing" schemes are possible, the best/simplest I found was to bucket on postcode.
| [reply] |
