‘rejecting I/O to offline device’

and

‘task jbd2/sda3-8 blocked for more than 120 seconds’

Scary stuff! Googling around found a lot of mention of serious hardware issues. Bummer! I figured it would take a while to figure it out. Probably need to get new drivers for the LSI SAS and build the new module.

Then I wondered if it was maybe the new kernel from the update. I went into /boot/grub/grub.conf and changed the default from 0 to 1, which selects the prior kernel.

Reboot. Problem solved. Going home.

I have the supermicro SuperServer 2026TT H6RF server. Each contains 4 nodes and each node has an onboard 6Gbps SAS controller, an LSI 2108. I wanted to install a manager for monitoring the drives and raid status.

Digging around I found:

ftp://ftp.supermicro.com/driver/SAS/LSI/2108/

which is the drivers and the MSM raid manager. I did not install the driver, centos 6 seems to utilize the SAS card just fine.

Upon unpacking the zip file:

ftp://ftp.supermicro.com/driver/SAS/LSI/2108/MSM/Linux/11.06-00/11.06-00.zip

I found in the disk directory the readme that has prerequisites for RHEL6. They list a bunch of rpms that need to be installed. I had the target packages installed via yum, but I got errors running the install script. It wasn’t clear what exactly I needed to install.

I think the pre-requisites are 32 bit libraries. At any rate after much fumbling around I found the MegaRaid manager worked when I installed the following yum packages:

yum install libstdc++.i686

yum install libXext.i686

yum install libXtst.i686

Good luck!

1) open to all who wish to join

2) Membership by request, anyone can request

3) Closed – Admin must add each User

FishBlab Fishing Groups allows Users to quickly group posts within a Group of interest. After sucessfully joining a Group Users have the option to set any post to be part of one of the Fishing Groups they are members of.

So there were actually 2 google plus one buttons and the tweet button was gone. I searched and searched for a solution but found none. I tried changing the order of the links and script to no avail. There is no company that has worse customer support then google, they just don’t care. No links for support or even a forum.

Finally, I saw this in the google documentation:

“Note: The +1 is a public action and must be performed on a public URL. If you attempt to +1 a URL that is not public, an error will occur.”

Well, yes my dev server is not public. So then I tried uploading to a public server and presto it was fixed.

So, it seems that if you add the google plusone button to a non public server – such as a dev server that is firewalled – an ‘error’ that may occur is to obscure/hide the twitter tweet button and overlay a second google plus one button. That seems like a pretty strange error. Perhaps they should say:

Note: The +1 link requires a public web server to function properly. Use on a non public server (such as development) may cause strange effects to the page.

Start with the chassis: The awesome Supermicro 2026TT. This badass in a 2U case has 4 nodes and can handle 24 2.5″ HDD’s. Each hot-swappable node has 2 sockets for Xeon processors 5600/5500, up to 196 Gig of ECC Ram and onboard LSI SAS 6g card to handle the max of six hot-swap drives.

Blades? We don’t need no … Who has the money for blades? This is an excellent startup box. Cost? For each node I configured: 48 G of ECC/REG RAM, 2 X 600G seagate HDD (to be added to later), 2X XEON E5620. I bought 2 of these servers for a total of 8 nodes. plus 2 extra drives for one of the boxes. $31,000 before tax. $31K for 8 rockin servers with 48G of RAM. This was on 12/2010.

Oh, and the redundant hot swap power supply consumes a max of 1400 watts which is 350 watts per server. Very low power for such beefy boxes.  After 6 months of operation these supermicro boxes have been flawless. Still running CENTOS 5.

Hot swap node

The default view shows a map driven view of all FishBlab Users. Change the map to see Users in other areas. Click on a User listing to see details, and choose the ‘Profile’ button to see all detail.

At any time a logged in User can view and edit their own profile. This includes account details, location, profile photo and more. This section will be changing so check for updates.

I decided I could afford the upfront costs of the servers and hosting and the flexibility would be a big advantage. By flexibility I mean to being able to have the exact hardware I needed. Like utilize seagate 2.5″ 10K 900GB HDD’s now that they are available, or move from 2.5″ 10K HDD’s to SSD’s as they become cheaper and better.

One thing I love is hardware. Fast, powerful and fun to fondle. In the end I could not resist, I rolled my own server/hosting solution. I spec’d out my own servers using supermicro servers chassis – which I will detail in another post. I selected the CPU’s, memory and HDD’s. I chose to skimp on CPU and HDD’s to reduce my upfront costs. I found a great local computer builder – http://www.centralcomputers.com – these guys are the best in the valley as far as I am concerned.

Did I mention I am cheap? I won’t even consider any solution from the likes of sun, oracle, ibm, hp, etc. I didn’t even look. I assume they won’t be able to provide a solution like I assembled for 2X or 4X the price. Besides, why waste time doing that when building the thing is so fun?

I chose Hurricaine Electric who I have used in the past. I searched around a bit and it seemed like everyone was selling around a MBit per second for $1000 or so with a rack. HE.net has a special of $600 for a full cabinet and 100MBit per sec. WOW!

The only problem was the power that came with that. 15 AMP. One of my servers maxed out would be bumping into that figure, and I wanted 2 servers with the headroom to add another 1-2 if I needed. Keep in mind these are 4 node servers. After some discussion I opted for a 208 volt drop that provides 20 amp. The 208 volt allows a lot less amperage draw, so I have the potential to add 2 more servers in the future if needed without incurring the cost of a new power drop. This makes it expensive – about $1000 a month. But for that I get a cabinet, 208 volt drop @ 20 amps and 100 MBit per sec. When the traffic comes I will be ready.

Fishing reports have been added, a great source for guides and charter captains to promote they’re businesses. Photos can be added to a report.

Spots are great for documenting the important areas used for bait, fishing or whatever. Spots need a description and are placed on the map as well. They can also have multiple photos.

A new section has been added to FishBlab: Photos. photo.fishblab.com is where users can quickly upload Fishing photos using a simple photo uploader. Once signed in the user can quickly add photos with optional captions and descriptions. Once uploaded the User is prompted to place the photo on the map using the standard FishBlab location placement popup. This section of the site will be enhanced soon with more user fishing content tools, check back soon!

The uploader uses a hidden iframe solution due to the fact that file upload is not supported via ajax. What a bummer! Ajax solutions everywhere but here. I needed to create a hidden iframe and use it as the target of the file upload form submission. When the upload is complete a javascript function is run to inform the User.

Showing the file upload progress is a pain. Once uploaded, the system goes through these steps: validate the file, save the original, make different sized copies, remove the original and finally copy the new files from the back servers to the front. I used rsync for the copy process which seems archaic but works.

An improvement on this process would be to return to the user immediately after the file has been uploaded. The problem with that is that the resized copies are not finished and the sync to the fronts is not finished. I would love to fork off a few processes (or threads) to handle the syncs and return to the user, but there is no forking in PHP in a web environment. Bummer! At some point I will need to revisit this code and come up with a more elegant solution.