Joe Junkin

Problems using YUI Panel

jjunkin — Fri, 28 Mar 2008 20:11:55 +0000

I use quite a few YUI panel objects as inline popups used for gathering and displaying information. Using the modal dialog option caused an IE error message: ‘A script on this page is causing Internet Explorer to run slowly …’ which caused me great worry until I discovered the cause and fix. I also had issues with KeyListener events being executed after a dialog had been closed. Finally, finding when to initialize the dialogs was a challenge.

I began getting the IE error message:

“Stop running the script? A script on this page is causing Internet
Explorer to run slowly. If it continues to run, your computer may become
unresponsive.”

Which was scary as I thought this was due to the amount of data I was loading into javscript arrays and my dynamic DOM events. The errors were continuous and appeared to occur after one or more of the YUI panels were displayed in modal mode. When I limited the amount of data being processed, the errors continued. I figured there must be something wrong with my code.

After many google searches on the error message, I found this link about a bug in the YUI panel in modal mode with complex pages:

http://sourceforge.net/tracker/index.php?func=detail&aid=1884118&group_id=165715&atid=836476

It appears that the YUI panel in modal mode is not scalable:

“The currently implementation, which attaches focus handlers (which blur) to all focus'able items on the page when modal, definitely has scalability issues.”

Thanks for letting me know guys! I just spent hours trying to figure out what was wrong with my code! It would be nice of them to put a note about this on the YUI webpage for the panel object.

The solution is to unsubscribe the listeners

dialog.showMaskEvent.unsubscribe();

dialog.hideMaskEvent.unsubscribe();

This degrades the modality a bit by still allowing the user to TAB into the page behind the mask and launch links, but they cannot click with the mouse. Good enough functionality for me.

I also had issues where events were being fired when the enter key was pressed. It seemed that the event from a closed dialog was being launched along with the currently open dialog’s key event. Unfortunately, it’s hard to pin down how this was fixed or why it was happening. I fiddled a bit and it seems fine now.

The final issue I had was when to initialize the panels. I started by using the YUI suggested method of using the YAHOO.util.Event.addListener method. The problem was that I already had a function running

I finally just called the routines in my

Strange server errors posting an Ajax form

jjunkin — Wed, 26 Mar 2008 21:50:25 +0000

I just wrestled with getting a form to post via ajax. I have done this before but had a bunch of issues crop up this time including strange intermitten server errors that only seemed to occur on Firefox. I resolved it by using the extremely handy YUI connection manager ’setForm’ function to process the data for me. I also found a problem when submitting a non-rails created form that caused an ctionController::InvalidAuthenticityToken error.

I was sending the form by using a javascript function to get each field, concatonate a url (using escape) and use the YUI connection manager with a GET method. This caused intermitten errors on firefox including one like ‘the server did not understand the request’ (or something like that). What was happening was the escape was creating a series of chars that triggered a mod_security rule. The rule was blocking access to the rails server.

ModSecurity: Access denied with code 400 (phase 2). Pattern match "%0[ad]” at REQUEST_URI. [id "950910"] [msg "HTTP Response Splitting Attack. Matched signature <%0a>"] [severity "ALERT"] [hostname "dev.removed.com"] [uri "/aSubmitContact?name=john%20johnson&email=john@removed.com&phone=123455&msg=this%20is%20a%20test%0Athsi%20is%20only%20a%20test&subject=general"]

I did some digging and discovered that YUI has a sweet function to fetch, prepare and send the form fields for me:

Connection Manager can automatically harvest HTML form data and prepare it for either a GET or POST request via the setForm method. When you call this method before initiating the transaction, Connection Manager constructs a GET querystring or a POST message from the form and submits it to the specified URL.

You simply hand the element to YAHOO.util.Connect.setForm(formObject, true); and the connection manager takes care of the rest. SWEET!

After that, another issue came up with rails. It seems that rails expects you to utilize it’s built-in form generator. When I submitted the form I built with javascript using the DOM, I got the error:

ActionController::InvalidAuthenticityToken

This appears to be due to a new feature in rails that attempts to stop cross-site-scripting by attaching an hidden identifier value to the form. I had to disable the feature for my javascript created form. To disable it I used:
protect_from_forgery :only => [:update, :delete, :create].
I realize that it is a nice feature, but it would be nice to see it as optional instead of default.

Geocoding solution

jjunkin — Mon, 17 Dec 2007 03:55:15 +0000

My Geocoding experience with Google and Yahoo became increasingly difficult and lengthy until I finally went looking for another solution. I found that solution with the Perl Module Geo::Coder::US. My final solution utilized Geo::Coder::US for most of the geocoding with Google and Yahoo helping on more difficult ones. The processing time for batch processing 20,000 real estate listings went from around 8 hours in the beginning down to about 1 minute in the end.

Batch geocoding 20,000+ real estate listings using the Google and Yahoo geocoders ended up taking about 8 hours to complete. At various points the process was cut off after to many requests, even after I placed a sleep(1) call before each transaction. I became very discouraged at the performance and went looking for another solution.

I investigated the Perl Module Geo::Coder::US and found it was extremely easy to setup and configure. The lookup process is nice because the module does some massaging of the input to standardize stuff like ‘av’ and ‘ave’. It takes the input as a single string like google and yahoo allow. The lookups are extremely fast. Without enabling the cache, Geo::Coder::US handled 80% of the lookups, google 13% and yahoo 7%. This process took about 18 minutes.

When the cache was enabled the process used the database for the majority of the lookups and the process took 1-3 minutes to complete.

Geocoding MLS data

jjunkin — Tue, 11 Dec 2007 21:43:18 +0000

The data provided by REIL does not contain geocoding which means that I will have to do this myself. I am using Perl to download and build the data so I have designed a Perl process to bulk geocode each of the addresses. I designed a caching system so that I only need to do any one lookup one time.

I use the Google and Yahoo geocoders which are free. Yahoo allows something like 5000 per day and Google allows around 15,000. Even before I approached these limits I was blocked because I was sending too many requests per second. To solve this issue I placed a ’sleep(1)’ line before each of the geocoder calls ensuring that I don’t call either geocoder more than once a second. Once I implemented this (and moved to a different IP) this seems to work fine.

My caching system uses a GOOD and BAD cache table. Values found in one or the other are returned instead of doing a Google/Yahoo geocode call. I have about 20,000 addresses in my ‘GOOD’ cache right now. I will create a cron job to expire these after some period of time, perhaps a couple of weeks.

Download continued

jjunkin — Thu, 22 Nov 2007 21:30:55 +0000

I have been doing continued refinement on the reil rets download. I decided to structure the download so that each ‘Class’ (residential, condo,etc) is downloaded into a separate table -as is. After the download is complete I run updates every 15 minutes. At first I attempted to place data into tables that would be accessed directly by the application, but I decided to change that process and create a final build table.

I added a new step in the process, I create a ‘combo’ table that combines all of the pertinent data from each of the classes/tables into one combined table. The reason for this is so a user can choose to search over any combination of the property types (i.e. residential and condos). Another benefit to this is that my combo table can be de-normalized (containing redundant lookup data) and contains only the data I need. There are a lot of extra empty columns in the dataset.

Download REIL Property data from scratch

jjunkin — Sat, 06 Oct 2007 02:48:19 +0000

My first activity was to download the property data. REIL has a bunch of property ‘classes’ such as RES (residential) and CON (Condominium). The classes will need to go into separate tables. So I designed my download process to take in a Resource (Property) and a Class (RES, CON, etc). Designing this way meant I could use the same routines for all classes of data.

The first step is to create the data table. At first I tried this manually, but later realized that I could just download the metadata and use this to create the table. So my first task was actually to download the metadata (for each class) and put it into a Â metadata table. A fringe benefit of this is that I now have information about each column of data such as datatype, long description, short description, length, etc. I will use the metadata to build the table now, but will also use it later for search forms and other system stuff.

After downloading the metadata for a particular class, I iterate through each record and build a CREATE TABLE statement. An obvious benefit here is that any column additions/modifications on REILâ€™s end will be incorporated each time the dataset is rebuilt. At the end of this process I have a fresh new up-to-date table ready to load with data.

There is a limit to the amount of data you can download in one batch. This limit becomes more constrained as you download more columns, and there are a lot of columns (177). You can use pagination, but the documentation mentions that modifications to the data between the time that the download begins and ends can cause issues. What happens if someone deletes or removes a property? This could screw up the pagination and something could be skipped.

I chose to download all of the propertyIDâ€™s at once, which the system allows since it is only one field. I load these into an array and proceed to break up the fetches of complete property data into chunks of 1000 or so. This allows the download of all of the data using an initial snapshot.

I download the property data in COMPACT mode and load it into a TEMP table. This allows me to build the complete dataset and verify it before (quickly) moving it into production. Â When the load is complete and the data count is verified, a RENAME TABLE moves the data into production very quickly.

The photos are next. I use a similar process for downloading the photos into a TEMP area. I download photo file names and other metadata into a Resource-Class specific table, and place the actual files in a hashed directory structure for quick lookup. The photo downloads are by far the longest part of the download process. I download the files to a TEMP directory and move the directory to production after validation.

I found that some files error out with a â€œNo Object Foundâ€ error message. Investigating further I found that sometimes these files become available later. So, I search for the error message and write a record to a â€˜photo errorâ€™ table which includes the propertyID and attempt count. Later I run a cron job to attempt to re-download these failed files.Â

Downloading REIL Data myself!

jjunkin — Sat, 01 Sep 2007 02:43:08 +0000

So, this is where I have ended up, writing the REIL data download from scratch. I am using Perl and imagine it would be difficult using something else. I want to be able to have one set of routines to download the data in whatever format I choose, and transfer the data to a MySQL database. Next I will need to download the images. I will need the data for all of the REIL property classes. Finally, I will need to have a routine that updates the data.

MRIS Conduit

jjunkin — Tue, 21 Aug 2007 02:25:15 +0000

MRIS Conduit is another free client for downloading data. In contrast to every other client I tried, MRIS Conduit was super simple, intuitive and worked the first time and every time. It downloads data to text/xml files and will also run updates using a built-in scheduler. It also allows you to define individual dataset queries, and combine them into sets that can be scheduled to run as needed. The drawback is that it lacks a direct-to-database import.

I would suggest starting out with this package. It allowed me to see the data and verify the correct parameters. It displays the query string used after you build it which is helpful. I got this to run on Linux as well as Windows. The easy to use interface is very well designed. I designed a Perl process to parse the data files it downloaded and put the information into a MySQL database.

In the end, I gave up and wrote my own download routine. There was just too much hassle of running different programs in synch. By writing a custom routine I was able to get exactly what I wanted the wayÂ I wanted. But MRIS Conduit was a helpful start and a usefulÂ tool forÂ data verification and exploration.Â Â

Using VieleRETS to download data

jjunkin — Sat, 11 Aug 2007 02:14:24 +0000

I spent a long time messing with VieleRETS to try and download data from REIL. The short answer is that I failed to get it to work. I tried to debug the PHP code but was unable to identify why it was failing. Had I needed to get this to work I would have joined the mail-list group and tried to get some help from there, but I never got that far.

This is a very complete package that should have done exactly what I needed. It allows you to enter your login credentials and then connect to the RETS compatible MLS, download metadata, download dataÂ and keep it updated.

Unfortunately, I was unable to get this to work.Â The system is notÂ very well documented at this point but certainly tries to do a lot. ItÂ did connect to REIL and downloaded some metadata.Â

My best guess is that it was trying to download property information without qualifying the ‘listing_status’ parameter, and I could find no place where I could specify this information. REIL requires the listing status (or at least some constraint). The data queries failed when I tried to run the update saying there was no data.

I expect this package to get better quickly as it seems to have regular updates and is actively developed.

Login to reil.com

jjunkin — Mon, 11 Jun 2007 20:51:20 +0000

REIL.com uses digest authentication. I am using perl to pull down the data so first I need to login to the reil.com server with proper authorization. I was unable to find a lot of documentation on how this process works. The usual method for exposing web services is via SOAP, but this service uses HTTP Digest Authentication. I have not used it much but it appears that the main benefit is not sending the password in clear text over a non-ssl connection. It is not as secure as SSL by any means but is better than a clear text transmission of the password.

The process appeared daunting at first, due to the lack of examples and documentation. The process was clear, attempt login to the server, the server rejects the login and requires login and finally the user sends validation. There are fields in the response header sent by the server that are used in the last step authentication.

What was not clear to me is how LWP::UserAgent to perform this feat. I could see that the mechanism was in LWP::Authen::Digest - authenticate, but i wasn’t sure how to access it. The docs mentioned overriding LWP:UserAgent’s get_basic_credentials sub to supply a password.

#!/usr/bin/perl package RequestAgent; @ISA = qw(LWP::UserAgent); use strict; use LWP::UserAgent; sub new { my $self = LWP::UserAgent::new(@_); $self->agent("lwp-request/$main::VERSION"); $self; } sub get_basic_credentials{ my($self, $realm, $uri) = @_; return('myUsername','myPassword'); }So once this module was created, I just used the following code and everything worked as it was supposed to.