David Currie

Shortly after moving my blog to an EC2 instance I received an ominous email stating:

We have noticed that one or more of your instances are running on a host degraded due to hardware failure.

The risk of your instances failing is increased at this point. We cannot determine the health of any applications running on the instances. We recommend that you launch replacement instances and start migrating to them.

After a bit of searching around it seemed that, as my instance is EBS backed, I should just need to stop it and restart it and, in all likelihood, it would move to another host. Sounded simple enough but the stop seemed to be taking forever. After downloading the command line tools I tried to force the stop but still no luck – except when I logged in again the following night it was finally showing as stopped. Unfortunately, when I tried to start it I received the following error: “Server.InternalError: Internal error on launch”.

I posted a question on the EC2 forum and, as you can see, was told that the root volume was in an “abnormal attachment state”. You’ll also see that my attempts to force a detach via the AWS Management Console appeared to fail or at least, the console thought the volume was still attached and therefore wouldn’t let me reattach it. Reverting to the command line utilities again allowed me to successfully detach and reattach the volume and then start the instance. Another delay whilst the DNS refresh took effect (perhaps I should try out an Elastic IP) and, about three days after I first hit stop, the site is finally back online. I’m putting it down as a learning experience!

When attempting to post the image for my last blog entry, it failed to be resized. When I logged in to the server to see what was up with ImageMagick, I didn’t appear to have permission to execute ls. Or ps. Or netstat… A quick Google suggested that these were the hallmarks of a rootkit attack. Unfortunately the files had been modified prior to the oldest Slicehost backup that I had. At this point I realised the server was still running Intrepid, limiting my chances of picking up packages to detect and remove rootkits. After a reboot of the server I discovered that I had lost all connectivity. Booting up a Slicehost rescue image I was able to retrieve all of the data I needed. Now to get things up and running again. Earlier in the year I had been playing around with a free micro-instance on EC2 and this seemed like the ideal opportunity to switch across. The instance is running Apache rather than nginx as on my Slicehost image. This needed a bit of tuning down to prevent segmentation faults. Everything seems to be running smoothly now. I just need to switch the DNS records away from Slicehost and then I’m done.