Geocoding solution
My Geocoding experience with Google and Yahoo became increasingly difficult and lengthy until I finally went looking for another solution. I found that solution with the Perl Module Geo::Coder::US. My final solution utilized Geo::Coder::US for most of the geocoding with Google and Yahoo helping on more difficult ones. The processing time for batch processing 20,000 real estate listings went from around 8 hours in the beginning down to about 1 minute in the end.
Geocoding MLS data
The data provided by REIL does not contain geocoding which means that I will have to do this myself. I am using Perl to download and build the data so I have designed a Perl process to bulk geocode each of the addresses. I designed a caching system so that I only need to do any one lookup one time.
Download continued
I have been doing continued refinement on the reil rets download. I decided to structure the download so that each ‘Class’ (residential, condo,etc) is downloaded into a separate table -as is. After the download is complete I run updates every 15 minutes. At first I attempted to place data into tables that would be accessed directly by the application, but I decided to change that process and create a final build table.
Download REIL Property data from scratch
My first activity was to download the property data. REIL has a bunch of property ‘classes’ such as RES (residential) and CON (Condominium). The classes will need to go into separate tables. So I designed my download process to take in a Resource (Property) and a Class (RES, CON, etc). Designing this way meant I could use the same routines for all classes of data.
Downloading REIL Data myself!
So, this is where I have ended up, writing the REIL data download from scratch. I am using Perl and imagine it would be difficult using something else. I want to be able to have one set of routines to download the data in whatever format I choose, and transfer the data to a MySQL database. Next I will need to download the images. I will need the data for all of the REIL property classes. Finally, I will need to have a routine that updates the data.
MRIS Conduit
MRIS Conduit is another free client for downloading data. In contrast to every other client I tried, MRIS Conduit was super simple, intuitive and worked the first time and every time. It downloads data to text/xml files and will also run updates using a built-in scheduler. It also allows you to define individual dataset queries, and combine them into sets that can be scheduled to run as needed. The drawback is that it lacks a direct-to-database import.
Using VieleRETS to download data
I spent a long time messing with VieleRETS to try and download data from REIL. The short answer is that I failed to get it to work. I tried to debug the PHP code but was unable to identify why it was failing. Had I needed to get this to work I would have joined the mail-list group and tried to get some help from there, but I never got that far.
Login to reil.com
REIL.com uses digest authentication. I am using perl to pull down the data so first I need to login to the reil.com server with proper authorization. I was unable to find a lot of documentation on how this process works. The usual method for exposing web services is via SOAP, but this service uses HTTP Digest Authentication. I have not used it much but it appears that the main benefit is not sending the password in clear text over a non-ssl connection. It is not as secure as SSL by any means but is better than a clear text transmission of the password.
Read more
Access to MLS reil.com
I have been working on a project to pull information from a local MLS service, reil.com. They provide MLS data for the San francisco Bay area such as Palo Alto and Menlo Park. I am building websites for some local Real Estate Agents that display current MLS properties in a google map. The data will be pulled from reil.com to a local database and syncronized at various periods throughout the day. The information will be displayed on a googlemap. The data will appear based on the current proximity of the map. So when the user drags the map the system will re-search the database give the new boundries.
Working with the Zillow API
We recently completed a website that displays property estimates using the data from http://zillow.com:
http://www.estimatepropertyvalues.com
Zillow provides excellent documentation on how to utilize their API’s which made is easy to configure. This application pulls zillow data in real time and displays it alongside and within a google map. Each plot on the map holds a tabbed view of information including property details and charts. The application is designed to operate using mostly ajax calls for faster responses. The main challenge here was how to display a large quantity of data in a clean, uncluttered fasion.