Batch geocoding 20,000+ real estate listings using the Google and Yahoo geocoders ended up taking about 8 hours to complete. At various points the process was cut off after to many requests, even after I placed a sleep(1) call before each transaction. I became very discouraged at the performance and went looking for another solution.

I investigated the Perl Module Geo::Coder::US and found it was extremely easy to setup and configure. The lookup process is nice because the module does some massaging of the input to standardize stuff like ‘av’ and ‘ave’. It takes the input as a single string like google and yahoo allow. The lookups are extremely fast. Without enabling the cache, Geo::Coder::US handled 80% of the lookups, google 13% and yahoo 7%. This process took about 18 minutes.

When the cache was enabled the process used the database for the majority of the lookups and the process took 1-3 minutes to complete.

I use the Google and Yahoo geocoders which are free. Yahoo allows something like 5000 per day and Google allows around 15,000. Even before I approached these limits I was blocked because I was sending too many requests per second. To solve this issue I placed a ’sleep(1)’ line before each of the geocoder calls ensuring that I don’t call either geocoder more than once a second. Once I implemented this (and moved to a different IP) this seems to work fine.

My caching system uses a GOOD and BAD cache table. Values found in one or the other are returned instead of doing a Google/Yahoo geocode call. I have about 20,000 addresses in my ‘GOOD’ cache right now. I will create a cron job to expire these after some period of time, perhaps a couple of weeks.