David Currie

Shortly after moving my blog to an EC2 instance I received an ominous email stating:

We have noticed that one or more of your instances are running on a host degraded due to hardware failure.

The risk of your instances failing is increased at this point. We cannot determine the health of any applications running on the instances. We recommend that you launch replacement instances and start migrating to them.

After a bit of searching around it seemed that, as my instance is EBS backed, I should just need to stop it and restart it and, in all likelihood, it would move to another host. Sounded simple enough but the stop seemed to be taking forever. After downloading the command line tools I tried to force the stop but still no luck – except when I logged in again the following night it was finally showing as stopped. Unfortunately, when I tried to start it I received the following error: “Server.InternalError: Internal error on launch”.

I posted a question on the EC2 forum and, as you can see, was told that the root volume was in an “abnormal attachment state”. You’ll also see that my attempts to force a detach via the AWS Management Console appeared to fail or at least, the console thought the volume was still attached and therefore wouldn’t let me reattach it. Reverting to the command line utilities again allowed me to successfully detach and reattach the volume and then start the instance. Another delay whilst the DNS refresh took effect (perhaps I should try out an Elastic IP) and, about three days after I first hit stop, the site is finally back online. I’m putting it down as a learning experience!

When attempting to post the image for my last blog entry, it failed to be resized. When I logged in to the server to see what was up with ImageMagick, I didn’t appear to have permission to execute ls. Or ps. Or netstat… A quick Google suggested that these were the hallmarks of a rootkit attack. Unfortunately the files had been modified prior to the oldest Slicehost backup that I had. At this point I realised the server was still running Intrepid, limiting my chances of picking up packages to detect and remove rootkits. After a reboot of the server I discovered that I had lost all connectivity. Booting up a Slicehost rescue image I was able to retrieve all of the data I needed. Now to get things up and running again. Earlier in the year I had been playing around with a free micro-instance on EC2 and this seemed like the ideal opportunity to switch across. The instance is running Apache rather than nginx as on my Slicehost image. This needed a bit of tuning down to prevent segmentation faults. Everything seems to be running smoothly now. I just need to switch the DNS records away from Slicehost and then I’m done.

Just to show that I do listen to you dear reader, I’ve followed up on Oliver’s suggestion and moved our e-mail over to Google Apps. The process was pretty painless: sign up for an admin account, verify ownership of the domains, setup users, modify the MX records and then configure our e-mail clients. A day or so delay whilst the DNS records replicate and everything is now working smoothly. POP3 and SMTP access is via SSL which is a bonus and means I no longer need to rely on my FastMail account to send when away from home. The spam filtering is excellent – I’ve even got a 12 year old e-mail address (a spam trap if ever there were one) coming back in to my main inbox again. Other than that, the only noticeable effect is that my incoming messages are no longer tarnished by a “Virus checked by PlusNet” footer. Oh, and of course there is the GMail web interface should I ever feel the urge…

Anyone trying to access this site yesterday will have just received an error message about lack of database connectivity. Fortunately, some twelve hours later, PlusNet have managed to restore database access and, on this occasion, without losing any data. Memo to self: backup tonight and increase urgency of search for new host.

I have used PlusNet as my ISP for nearly a year now and my disillusionment is now pretty much complete. At the time I moved over they came highly recommended and, unbelievably, their website is currently trumpeting that they were top of a customer satisfaction survey by uswitch.com.
(more…)