Batch geocoding 20,000+ real estate listings using the Google and Yahoo geocoders ended up taking about 8 hours to complete. At various points the process was cut off after to many requests, even after I placed a sleep(1) call before each transaction. I became very discouraged at the performance and went looking for another solution.

I investigated the Perl Module Geo::Coder::US and found it was extremely easy to setup and configure. The lookup process is nice because the module does some massaging of the input to standardize stuff like ‘av’ and ‘ave’. It takes the input as a single string like google and yahoo allow. The lookups are extremely fast. Without enabling the cache, Geo::Coder::US handled 80% of the lookups, google 13% and yahoo 7%. This process took about 18 minutes.

When the cache was enabled the process used the database for the majority of the lookups and the process took 1-3 minutes to complete.

I use the Google and Yahoo geocoders which are free. Yahoo allows something like 5000 per day and Google allows around 15,000. Even before I approached these limits I was blocked because I was sending too many requests per second. To solve this issue I placed a ’sleep(1)’ line before each of the geocoder calls ensuring that I don’t call either geocoder more than once a second. Once I implemented this (and moved to a different IP) this seems to work fine.

My caching system uses a GOOD and BAD cache table. Values found in one or the other are returned instead of doing a Google/Yahoo geocode call. I have about 20,000 addresses in my ‘GOOD’ cache right now. I will create a cron job to expire these after some period of time, perhaps a couple of weeks.

I added a new step in the process, I create a ‘combo’ table that combines all of the pertinent data from each of the classes/tables into one combined table. The reason for this is so a user can choose to search over any combination of the property types (i.e. residential and condos). Another benefit to this is that my combo table can be de-normalized (containing redundant lookup data) and contains only the data I need. There are a lot of extra empty columns in the dataset.

The first step is to create the data table. At first I tried this manually, but later realized that I could just download the metadata and use this to create the table. So my first task was actually to download the metadata (for each class) and put it into a  metadata table. A fringe benefit of this is that I now have information about each column of data such as datatype, long description, short description, length, etc. I will use the metadata to build the table now, but will also use it later for search forms and other system stuff.

After downloading the metadata for a particular class, I iterate through each record and build a CREATE TABLE statement. An obvious benefit here is that any column additions/modifications on REIL’s end will be incorporated each time the dataset is rebuilt. At the end of this process I have a fresh new up-to-date table ready to load with data.

There is a limit to the amount of data you can download in one batch. This limit becomes more constrained as you download more columns, and there are a lot of columns (177). You can use pagination, but the documentation mentions that modifications to the data between the time that the download begins and ends can cause issues. What happens if someone deletes or removes a property? This could screw up the pagination and something could be skipped.

I chose to download all of the propertyID’s at once, which the system allows since it is only one field. I load these into an array and proceed to break up the fetches of complete property data into chunks of 1000 or so. This allows the download of all of the data using an initial snapshot.

I download the property data in COMPACT mode and load it into a TEMP table. This allows me to build the complete dataset and verify it before (quickly) moving it into production.  When the load is complete and the data count is verified, a RENAME TABLE moves the data into production very quickly.

The photos are next. I use a similar process for downloading the photos into a TEMP area. I download photo file names and other metadata into a Resource-Class specific table, and place the actual files in a hashed directory structure for quick lookup. The photo downloads are by far the longest part of the download process. I download the files to a TEMP directory and move the directory to production after validation.

I found that some files error out with a “No Object Found” error message. Investigating further I found that sometimes these files become available later. So, I search for the error message and write a record to a ‘photo error’ table which includes the propertyID and attempt count. Later I run a cron job to attempt to re-download these failed files. 

This is a very complete package that should have done exactly what I needed. It allows you to enter your login credentials and then connect to the RETS compatible MLS, download metadata, download data and keep it updated.

Unfortunately, I was unable to get this to work. The system is not very well documented at this point but certainly tries to do a lot. It did connect to REIL and downloaded some metadata. 

My best guess is that it was trying to download property information without qualifying the ‘listing_status’ parameter, and I could find no place where I could specify this information. REIL requires the listing status (or at least some constraint). The data queries failed when I tried to run the update saying there was no data.

I expect this package to get better quickly as it seems to have regular updates and is actively developed.

What was not clear to me is how LWP::UserAgent to perform this feat. I could see that the mechanism was in LWP::Authen::Digest – authenticate, but i wasn’t sure how to access it. The docs mentioned overriding LWP:UserAgent’s get_basic_credentials sub to supply a password.


#!/usr/bin/perl
package RequestAgent;
@ISA = qw(LWP::UserAgent);
use strict;
use LWP::UserAgent;
sub new {
my $self = LWP::UserAgent::new(@_);
$self->agent("lwp-request/$main::VERSION");
$self;
}
sub get_basic_credentials{
my($self, $realm, $uri) = @_;
return('myUsername','myPassword');
}So once this module was created, I just used the following code and everything worked as it was supposed to.