‘rejecting I/O to offline device’

and

‘task jbd2/sda3-8 blocked for more than 120 seconds’

Scary stuff! Googling around found a lot of mention of serious hardware issues. Bummer! I figured it would take a while to figure it out. Probably need to get new drivers for the LSI SAS and build the new module.

Then I wondered if it was maybe the new kernel from the update. I went into /boot/grub/grub.conf and changed the default from 0 to 1, which selects the prior kernel.

Reboot. Problem solved. Going home.

Start with the chassis: The awesome Supermicro 2026TT. This badass in a 2U case has 4 nodes and can handle 24 2.5″ HDD’s. Each hot-swappable node has 2 sockets for Xeon processors 5600/5500, up to 196 Gig of ECC Ram and onboard LSI SAS 6g card to handle the max of six hot-swap drives.

Blades? We don’t need no … Who has the money for blades? This is an excellent startup box. Cost? For each node I configured: 48 G of ECC/REG RAM, 2 X 600G seagate HDD (to be added to later), 2X XEON E5620. I bought 2 of these servers for a total of 8 nodes. plus 2 extra drives for one of the boxes. $31,000 before tax. $31K for 8 rockin servers with 48G of RAM. This was on 12/2010.

Oh, and the redundant hot swap power supply consumes a max of 1400 watts which is 350 watts per server. Very low power for such beefy boxes.  After 6 months of operation these supermicro boxes have been flawless. Still running CENTOS 5.

Hot swap node

I decided I could afford the upfront costs of the servers and hosting and the flexibility would be a big advantage. By flexibility I mean to being able to have the exact hardware I needed. Like utilize seagate 2.5″ 10K 900GB HDD’s now that they are available, or move from 2.5″ 10K HDD’s to SSD’s as they become cheaper and better.

One thing I love is hardware. Fast, powerful and fun to fondle. In the end I could not resist, I rolled my own server/hosting solution. I spec’d out my own servers using supermicro servers chassis – which I will detail in another post. I selected the CPU’s, memory and HDD’s. I chose to skimp on CPU and HDD’s to reduce my upfront costs. I found a great local computer builder – http://www.centralcomputers.com – these guys are the best in the valley as far as I am concerned.

Did I mention I am cheap? I won’t even consider any solution from the likes of sun, oracle, ibm, hp, etc. I didn’t even look. I assume they won’t be able to provide a solution like I assembled for 2X or 4X the price. Besides, why waste time doing that when building the thing is so fun?

I chose Hurricaine Electric who I have used in the past. I searched around a bit and it seemed like everyone was selling around a MBit per second for $1000 or so with a rack. HE.net has a special of $600 for a full cabinet and 100MBit per sec. WOW!

The only problem was the power that came with that. 15 AMP. One of my servers maxed out would be bumping into that figure, and I wanted 2 servers with the headroom to add another 1-2 if I needed. Keep in mind these are 4 node servers. After some discussion I opted for a 208 volt drop that provides 20 amp. The 208 volt allows a lot less amperage draw, so I have the potential to add 2 more servers in the future if needed without incurring the cost of a new power drop. This makes it expensive – about $1000 a month. But for that I get a cabinet, 208 volt drop @ 20 amps and 100 MBit per sec. When the traffic comes I will be ready.

It uses doctrine which appears to be a mature SQL object relational mapper. It’s simple to use and has provided all of the flexibility I have needed so far.

Fishblab.com contains a map based toolset for discussing, reporting and analyzing fish species and fishing areas. There are three areas of FishBlab:

www.fishblab.com contains listings of the areas and species of Fish contained within the application. It begins at the highest level, states, cities and Fish species. The User can drill down within the areas to see more detail such as available fish and fishing spots. This part of the application ends with the user sent to the one of the two main Fishblab.com applications: by fish or by area.

area.fishblab.com is a map driven view of fishing driven by the current location. When the users enters a search area, the system zooms in on that area and shows all available fish species as well as fishing areas.

fish.fishblab.com is a map driven view of fishing by a single species of fish. When the user enters a fish name the system can display all areas of the range as well as discussion and reports for the target species only.

It is now safe to say that the issue has been resolved with the switch from centos 5 to rhel 6. This was for a php application server running php 5.3 connecting to a separate mysql server. I had used the Webtatic yum builds. The issue has not occurred again. If you are having the same problem I suggest taking a look at your php end.

I have 8 machines on a site. RHEL wants $349 per server with no support. For the privilege of 3 years of updates I would have to shell out $8376.00 for a bunch of FREE FUCKING SOFTWARE. Do you think I am fucking stupid? If it were not for centos, I would have long ago bailed to Ubuntu.

I fucking hate per-socket pricing. I mean, that’s just fucking greedy. I’m serously. That’s the shit I expect from microsoft and oracle – the really, really greedy fucks. Who buys oracle? You gotta be a really stupid fuck to buy oracle. Come on, google, yahoo and many others use mysql and it works just fine.

But back to per socket pricing on RHEL for a 4 socket setup. $1598 x  8 machines x 3 years =  $38, 352. FOR A BUNCH OF FREE SOFTWARE? What kind of moron pays that when I can get the same shit for free? Support? Google is my support contract. What a fucking joke.

Look, I like what RedHat does, they do give back. But prices like that? FUCK YOU REDHAT!

I was getting an intermittenet error on a CENTOS 5 PHP 5.3 server. Hard to recreate as it occurred every few hours. This is what would show in the log: pdo connection error: sqlstate[ 2013  lost connection to mysql server at ‘reading initial communication packet’, system error: 113. 113 is ‘no route to host’, which means it is some goddam networking nightmare problem!

I have been ripping all the hair out of my head trying to fix this FUCKING error for a week straight! What a goddam nightmare! I have tried everything to fix it. I tested and reconfigured my network countless times. I tweaked mysql, and finally downloaded the binary rpms from mysql and did a fresh install.  I poured over my PHP install.

But most of all, I searched google for days looking for discussion about this issue. I found lots of stuff but no difinitive answers. I also found the error popping up on sites all over the web – it is clearly occurring quite a bit.

What happens is the site seems just fine. I hit it repeatedly using the browser, and run thousands of siege tests – but no error. Then out of no where, maybe every hour or 2 I get the error again.

It was easy to blame mysql, as this bug discusses. But after I completely reinstalled it from the mysql rpms I had to rule that out. Either my network was flaky or the PHP server was causing it. One interesting thing is that I have a mirrored set of servers, a three tier front, back and database setup. The error was happening accross both setups, so it was clearly not a one off problem.

Oh so you are looking for a solution? Well … I can tell you what I found.

I am using symfony 1.4 which requires PHP 5.3. I have become old and lazy so I no longer have the energy to roll my own (compile) binaries. I searched around and found a site that had easy to install rpms using yum. The site is Webtatic. The author had all of the binaries I needed for PHP 5.3 so I just followed the instructions and BINGO I had a sweet PHP 5.3 setup!

Now, are these 3rd party rpms the cause of the problem or is it something in CENTOS 5? I really don’t know. But I do know that when I simply installed RHEL 6 and all the yummy PHP 5.3 stuff that comes with it, all was well. The error is gone – at least since I made the switch. If it comes back again, I will update this post.

Setup was trivial, and the big win is out of the box PHP 5.3. I have been chasing bugs with a PHP 5.3 CENTOS 5 installation for over a week (more on that later) , so this is a welcome change.  APC,PDO,MYSQL – just check the boxes and you are done. And the graphical installation didn’t crash on me like I am used to!

So, I will fiddle with this for the 30 day eval, then I will be expecting to dump it for CENTOS 6.