I use quite a few YUI panel objects as inline popups used for gathering and displaying information. Using the modal dialog option caused an IE error message: ‘A script on this page is causing Internet Explorer to run slowly …’ which caused me great worry until I discovered the cause and fix. I also had issues with KeyListener events being executed after a dialog had been closed. Finally, finding when to initialize the dialogs was a challenge.
I just wrestled with getting a form to post via ajax. I have done this before but had a bunch of issues crop up this time including strange intermitten server errors that only seemed to occur on Firefox. I resolved it by using the extremely handy YUI connection manager ’setForm’ function to process the data for me. I also found a problem when submitting a non-rails created form that caused an ctionController::InvalidAuthenticityToken error.
My Geocoding experience with Google and Yahoo became increasingly difficult and lengthy until I finally went looking for another solution. I found that solution with the Perl Module Geo::Coder::US. My final solution utilized Geo::Coder::US for most of the geocoding with Google and Yahoo helping on more difficult ones. The processing time for batch processing 20,000 real estate listings went from around 8 hours in the beginning down to about 1 minute in the end.
The data provided by REIL does not contain geocoding which means that I will have to do this myself. I am using Perl to download and build the data so I have designed a Perl process to bulk geocode each of the addresses. I designed a caching system so that I only need to do any one lookup one time.
I have been doing continued refinement on the reil rets download. I decided to structure the download so that each ‘Class’ (residential, condo,etc) is downloaded into a separate table -as is. After the download is complete I run updates every 15 minutes. At first I attempted to place data into tables that would be accessed directly by the application, but I decided to change that process and create a final build table.
My first activity was to download the property data. REIL has a bunch of property ‘classes’ such as RES (residential) and CON (Condominium). The classes will need to go into separate tables. So I designed my download process to take in a Resource (Property) and a Class (RES, CON, etc). Designing this way meant I could use the same routines for all classes of data.
So, this is where I have ended up, writing the REIL data download from scratch. I am using Perl and imagine it would be difficult using something else. I want to be able to have one set of routines to download the data in whatever format I choose, and transfer the data to a MySQL database. Next I will need to download the images. I will need the data for all of the REIL property classes. Finally, I will need to have a routine that updates the data.
MRIS Conduit is another free client for downloading data. In contrast to every other client I tried, MRIS Conduit was super simple, intuitive and worked the first time and every time. It downloads data to text/xml files and will also run updates using a built-in scheduler. It also allows you to define individual dataset queries, and combine them into sets that can be scheduled to run as needed. The drawback is that it lacks a direct-to-database import.
I spent a long time messing with VieleRETS to try and download data from REIL. The short answer is that I failed to get it to work. I tried to debug the PHP code but was unable to identify why it was failing. Had I needed to get this to work I would have joined the mail-list group and tried to get some help from there, but I never got that far.
REIL.com uses digest authentication. I am using perl to pull down the data so first I need to login to the reil.com server with proper authorization. I was unable to find a lot of documentation on how this process works. The usual method for exposing web services is via SOAP, but this service uses HTTP Digest Authentication. I have not used it much but it appears that the main benefit is not sending the password in clear text over a non-ssl connection. It is not as secure as SSL by any means but is better than a clear text transmission of the password.
Read the rest of this entry »