David Currie

This blog has been a bit neglected for the last few years. I miss the opportunity to reflect on something I’ve done and write up those thoughts. We’ll see whether this is a one-off or the start of something new!

The first task was to make sure everything on the site was up-to-date. WordPress does a pretty good job of automatically applying security fixes but the Ubuntu VPS needed an upgrade. The update to Jammy went smoothly enough but attempting to access the site showed the WordPress PHP source. The enabled modules for Apache showed a couple of broken symlinks to PHP 7. After enabling those for PHP 8.1, I saw a WordPress error page: There has been a critical error on this website.

The WordPress PHP compatibility matrix indicates that there are still exceptions with PHP 8 versions. Time to get PHP 7 back…

Then re-enable the PHP 7 Apache modules:

With the site rendering again, I thought I was done but on posting this entry the dreaded critical error reappeared. Looking again at the Apache error logs, /var/log/apache2/error.log revealed errors in lightbox-plus and crayon-syntax-highlighter of the form Compilation failed: invalid range in character class. From PHP 7.3, hyphens need to be escaped in regular expressions. I could have rolled the PHP version back further but decided to patch the offending files. (I probably need to review the plugins in use on the site and remove those that are no longer supported.)

And, finally, we’re back in business!

Shortly after moving my blog to an EC2 instance I received an ominous email stating:

We have noticed that one or more of your instances are running on a host degraded due to hardware failure.

The risk of your instances failing is increased at this point. We cannot determine the health of any applications running on the instances. We recommend that you launch replacement instances and start migrating to them.

After a bit of searching around it seemed that, as my instance is EBS backed, I should just need to stop it and restart it and, in all likelihood, it would move to another host. Sounded simple enough but the stop seemed to be taking forever. After downloading the command line tools I tried to force the stop but still no luck – except when I logged in again the following night it was finally showing as stopped. Unfortunately, when I tried to start it I received the following error: “Server.InternalError: Internal error on launch”.

I posted a question on the EC2 forum and, as you can see, was told that the root volume was in an “abnormal attachment state”. You’ll also see that my attempts to force a detach via the AWS Management Console appeared to fail or at least, the console thought the volume was still attached and therefore wouldn’t let me reattach it. Reverting to the command line utilities again allowed me to successfully detach and reattach the volume and then start the instance. Another delay whilst the DNS refresh took effect (perhaps I should try out an Elastic IP) and, about three days after I first hit stop, the site is finally back online. I’m putting it down as a learning experience!

When attempting to post the image for my last blog entry, it failed to be resized. When I logged in to the server to see what was up with ImageMagick, I didn’t appear to have permission to execute ls. Or ps. Or netstat… A quick Google suggested that these were the hallmarks of a rootkit attack. Unfortunately the files had been modified prior to the oldest Slicehost backup that I had. At this point I realised the server was still running Intrepid, limiting my chances of picking up packages to detect and remove rootkits. After a reboot of the server I discovered that I had lost all connectivity. Booting up a Slicehost rescue image I was able to retrieve all of the data I needed. Now to get things up and running again. Earlier in the year I had been playing around with a free micro-instance on EC2 and this seemed like the ideal opportunity to switch across. The instance is running Apache rather than nginx as on my Slicehost image. This needed a bit of tuning down to prevent segmentation faults. Everything seems to be running smoothly now. I just need to switch the DNS records away from Slicehost and then I’m done.

Just to show that I do listen to you dear reader, I’ve followed up on Oliver’s suggestion and moved our e-mail over to Google Apps. The process was pretty painless: sign up for an admin account, verify ownership of the domains, setup users, modify the MX records and then configure our e-mail clients. A day or so delay whilst the DNS records replicate and everything is now working smoothly. POP3 and SMTP access is via SSL which is a bonus and means I no longer need to rely on my FastMail account to send when away from home. The spam filtering is excellent – I’ve even got a 12 year old e-mail address (a spam trap if ever there were one) coming back in to my main inbox again. Other than that, the only noticeable effect is that my incoming messages are no longer tarnished by a “Virus checked by PlusNet” footer. Oh, and of course there is the GMail web interface should I ever feel the urge…