‘rejecting I/O to offline device’

and

‘task jbd2/sda3-8 blocked for more than 120 seconds’

Scary stuff! Googling around found a lot of mention of serious hardware issues. Bummer! I figured it would take a while to figure it out. Probably need to get new drivers for the LSI SAS and build the new module.

Then I wondered if it was maybe the new kernel from the update. I went into /boot/grub/grub.conf and changed the default from 0 to 1, which selects the prior kernel.

Reboot. Problem solved. Going home.

I have the supermicro SuperServer 2026TT H6RF server. Each contains 4 nodes and each node has an onboard 6Gbps SAS controller, an LSI 2108. I wanted to install a manager for monitoring the drives and raid status.

Digging around I found:

ftp://ftp.supermicro.com/driver/SAS/LSI/2108/

which is the drivers and the MSM raid manager. I did not install the driver, centos 6 seems to utilize the SAS card just fine.

Upon unpacking the zip file:

ftp://ftp.supermicro.com/driver/SAS/LSI/2108/MSM/Linux/11.06-00/11.06-00.zip

I found in the disk directory the readme that has prerequisites for RHEL6. They list a bunch of rpms that need to be installed. I had the target packages installed via yum, but I got errors running the install script. It wasn’t clear what exactly I needed to install.

I think the pre-requisites are 32 bit libraries. At any rate after much fumbling around I found the MegaRaid manager worked when I installed the following yum packages:

yum install libstdc++.i686

yum install libXext.i686

yum install libXtst.i686

Good luck!

Start with the chassis: The awesome Supermicro 2026TT. This badass in a 2U case has 4 nodes and can handle 24 2.5″ HDD’s. Each hot-swappable node has 2 sockets for Xeon processors 5600/5500, up to 196 Gig of ECC Ram and onboard LSI SAS 6g card to handle the max of six hot-swap drives.

Blades? We don’t need no … Who has the money for blades? This is an excellent startup box. Cost? For each node I configured: 48 G of ECC/REG RAM, 2 X 600G seagate HDD (to be added to later), 2X XEON E5620. I bought 2 of these servers for a total of 8 nodes. plus 2 extra drives for one of the boxes. $31,000 before tax. $31K for 8 rockin servers with 48G of RAM. This was on 12/2010.

Oh, and the redundant hot swap power supply consumes a max of 1400 watts which is 350 watts per server. Very low power for such beefy boxes.  After 6 months of operation these supermicro boxes have been flawless. Still running CENTOS 5.

Hot swap node

The problem is that it does not work out of the box for more than one domain. If you want to use domain1.com, domain2.com, domain3.com, etc it requires some hacking. I did set it up and it was not a big deal, but I was not super happy with it. It requires manually adding data directly to the mysql tables. The added domains do not appear under the main wordpress MU install.

We decided to forgo the hack and install WordPress MU where clients will be allowing users to setup many blogs (under one domain name)  and for each user with their own domain name, a standalone WordPress install.

This obviously adds to our maintenance but seems like a reasonable solution. If WordPress MU allows out-of-the-box support for multiple domains we will probably switch to it in the future as a single point solution.

UPDATE:
Wordpress MU does not work properly with the gallery2 plugin. This means that it will not work for many of the blogs I host. There are some hacks around this but it appears messy.

So our plans with WordPress will be to utilize wordpress mu for free or trial blogs, or for clients who want to offer sitewide wordpress services. There are too many problems with trying to using a single wordpress install for all clients.

I set up apache virtualhosts for each blog. The web address that is configured in the virtualhost should match the address in WordPress. This is because wordpress stores the hostname in the database. ServerAlias’s should probably be avoided by using redirects to a single host instead.


<VirtualHost>
ServerName joe.junkin.com
ServerAdmin admin@spammenot.com
DocumentRoot /home/apacheuser/html/blog/jjunkin
ErrorLog /var/log/httpd/wp__error.log
CustomLog /var/log/httpd/wp_access.log common
DirectoryIndex index.php
<Directory>
# would allow .htaccess to be read
# AllowOverride FileInfo Options
# instead. this disallows .htaccess
AllowOverride None
# the stuff needed for permalinks
RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</Directory>
</VirtualHost>

When I tried to switch to date and named based permalinks I received a webserver crash. This was because in my VirtualHost AllowOverride was set to None or Options. I needed to add FileInfo to allow Apache to process the local .htaccess. After I just decided to move the necessary code to the VirtualHost section within Apache and changed the AllowOverride back to None (as above).

At any rate Postfix handily rejects these messages quickly, short circuiting the spammers. This saves a lot of load on the mail server. It never receives the message, once it determines the message is targeted for a non-existant account it rejects the delivery attempt.

In qmail, this was a major pain in the ass and was poorly documented. There is a patch called ‘validrcptto’ but it was a pain to get operational. Everything in qmail is a patch and a pain.

Postfix is light years ahead of qmail. For starters, it’s ready to plug right into FC6 as a yum update. No need to download and compile a long chain of sources as in qmail. And there is even a handy little switcher in fedora that switches the system from sendmail to postfix ‘system-switch-mail-nox’. This takes care of starting and stopping the process on startup and other stuff. Nice and easy.

The config file for postfix is easy enough. /etc/postfix/main.cf holds most of the config stuff. I also utilized the virtual alias domains and accounts features with two other files which are utilized as hash lookups.

One interesting thing: In the last year or so we began to be targeted with massive amounts of spam being sent to nonexisitant accounts at all of our domains (we have about 10). My haunch is that the spammers were actually counting on the bounces that our server produced to send spam. I tried a few times to configure the validrcptto patch to stop this but was unable to get it to work. The amount of bounces went from almost zero to hundreds or thousands a day.

Good riddence to qmail!