Server Downtime (Sept 29, 2006)

As lots of you probably noticed, the PortableApps.com server was inaccessible today from around 3:30AM to 5:30PM New York time. This was due to an odd chain of events. But the issue has now been corrected and the server should be operating even faster than it was previously. For those interested in the technical details, read on...

Technical Details

As I'd mentioned previously, one of the hard drives on the server failed. This isn't normally a huge deal as the server uses a RAID5 drive array (multiple hard drives acting as one so that if one fails, the server keeps on going and a new drive can be popped in). While the drive was being replaced this morning, the box needed to be rebooted as a RAID BIOS update was applied. While shutting down, mySQL apparently failed to properly close 3 tables used by Drupal: cache, accesslog and sessions. When the box was brought back up with the new drive in place, the RAID array rebuilt itself as expected. mySQL on the other hand attempted to recover from a crash and failed to recover the 3 tables that weren't closed properly. So, whenever someone attempted to access the site... they either got nothing... or a page (or 1/2 a page) after about 2 minutes.

Misdiagnosis Due To Popularity

While attempting to diagnose the issue, RackSpace's techs thought the server may be under a DDOS attack. It turns out this was simply due to the fact that the connections hung due to the table corruption... and the fact that the server is so busy that, as soon as Apache would come back up it was flooded with connection attempts. Unfortunately, this misdiagnosis wound up costing a bit of time as we attempted to setup rules in iptables and things of that nature.

A Solution... Now Better, Stronger, Faster

While working through the issue with one of the techs in one of the many hourlong phone calls I was on today, we were able to get the load down enough so that the site came back. The RackSpace tech accomplished by limiting connections and by enabling query cache in mySQL (which had been overlooked previously). The site was back, but logins didn't work and the behavior of the URLs when attempting them lead me to believe there was some database corruption. Now that the site was accessible enough to log into phpMyAdmin I checked out the database and, sure enough, corrupt tables. I manually recreated each of them and we were back in business.

An interesting side effect of this is that the site is now (correctly) using mySQL query cache (set at 200MBs to be precise) so you should notice more "pop" while viewing pages on the site.

Well, that's the long and short of it... ok, the long of it. Sorry if this interfered with anyone else's day. I know it destroyed mine. (The last thing I needed was another day without working on paying work.) I think I need a nap.

Regards,
John