Archive for the ‘Web’ Category

Website backup to pCloud

Wednesday, January 30th, 2019

Another SOC website related posting – this time on the subject of backup. The website is backed up by the club’s current hosting provider (Krystal – who, a year in, I can highly recommend) but I was informed that the club had bought a large quantity of cloud storage for the purpose of storing its map archive and, for belt and braces, it made sense to also include backups of the website there.

As it turned out, the cloud storage was courtesy of pCloud who are best described as a Dropbox clone i.e. the expected interaction patterns are via the web UI, mobile, or sync from the desktop app. A quick search turned up rclone which describes itself as “rsync for cloud storage” and, amongst the list of supported backends, includes pCloud.

Install on hosting provider was straightforward. The configuration process is interactive (opening a browser to log in to pCloud) but the docs also cover how to create the configuration on one machine and copy them across to another. A copy is then as simple as:

I started out looking to use drush arb to create a backup but, as the same hosting is used for a WordPress site, it was easiest in the end just to write a script using tar and mysqldump to create the archive of the file system and database tables. This is then triggered nightly on a cron job. Each backup is around 0.5GB so I wasn’t too concerned about incremental backup and, with 2 TB of storage to play with, it will be a while before the question of cleaning up old backups comes back to haunt me!

Drupal 8 Migration

Monday, January 28th, 2019

For my sins, I have now been involved in the management of our orienteering club’s current website for over 10 years now. Back then, we wanted to make it as easy as possible for club officials and members to contribute content and, after evaluating WordPress, Joomla! and Drupal, we went with Drupal as our Content Management System. The extensibility of Drupal makes it immensely powerful but, as with many open source projects, the rich ecosystem of contributed modules can be both a blessing and a curse.

Although the details have been long forgotten, I do remember that the move from Drupal 6 to 7 was a painful one and so, despite it being over three years since Drupal 8 was released, I was in no rush to migrate. In the end, it was a security vulnerability in one of the modules that wasn’t going to be addressed in v7 that precipitated the move.

The major changes in core Drupal have seemingly been too much for many module contributors to make the move. An initial assessment wasn’t particularly promising: of fifty-five non-core modules the current site had installed, five were no-longer needed in Drupal 8, six had GA v8 versions and a further fourteen had beta versions available. A migration estimate site put the effort involved at several weeks worth and, in the end, it probably wasn’t far off!

My first task was to slim down the number of modules installed. Many weren’t actively in use any more (e.g. content_access and views_data_export) and others had simple replacements which had easier migration paths (e.g. swapping out timefield for a simple text field). Ironically, the module with the security flaw was one of those that I disabled but, having started down this path, I was determined to complete a migration.

It was then time to start the actual migration. Thankfully the process now involves setting up a parallel site as it would still be weeks before I had anything that was approaching usable. One of the issues was that no private file path was set up during the migration. Another, that the migrated text formats were using a handler that no longer existed. Opening and resaving them fixed that problem. Another of the random error messages required manually modifying the database to remove the upload field from entity.definitions.bundle_field_map in the drup_key_value table (go figure).

The site makes extensive use of custom content types and views which are finally a part of core Drupal. Views are not part of the default migration though, and, in the end, I just recreated them manually. The same was true of all the patterns for pathauto.

At this point, with the styling also re-introduced, the site was ready to go live again but there were still problems waiting to be found. One was that, what used to appear as a date field, now appeared as a datetime field in forms. In the end, I decided to test out the new REST capabilities to export the contents of the field and reimport into a new field with the correct type. The only catch here was that there is no querying capability in the REST API so it was necessary to create a JSON-rendered view that listed the required nodes in order to retrieve their ids so that they could then be processed one-by-one. The rest was just a short bash script using curl and jq.

Hopefully, the migration can now be considered complete. The site now uses relatively few custom modules which is, undoubtedly, a good thing for future stability. If the move to Drupal 9 looks anywhere near as painful though, I now know how to extract the entire site content so maybe it will be time to revisit the CMS landscape. It would hate to think that I’ll still be debugging PHP errors in another ten years time!

ES2015 in Production

Thursday, April 21st, 2016

Bård Hovde gave tonight’s Developer South Coast presentation on the subject of “ES2015 in Production” (or “ES6 in Production” if you must). You can find the slides here with the source for the presentation over on Bård’s GitHub account. He did a great job of making the subject matter entertaining. Beyond being able to say goodbye to all of that boilerplate, my main takeaway was the use of Babel for transpiling ES2015 into ES5, so no excuses about waiting for browser compatibility! The Babel site also has a nice overview of ECMAScript 2015 features.

Back online

Sunday, March 20th, 2011

Shortly after moving my blog to an EC2 instance I received an ominous email stating:

We have noticed that one or more of your instances are running on a host degraded due to hardware failure.

The risk of your instances failing is increased at this point. We cannot determine the health of any applications running on the instances. We recommend that you launch replacement instances and start migrating to them.

After a bit of searching around it seemed that, as my instance is EBS backed, I should just need to stop it and restart it and, in all likelihood, it would move to another host. Sounded simple enough but the stop seemed to be taking forever. After downloading the command line tools I tried to force the stop but still no luck – except when I logged in again the following night it was finally showing as stopped. Unfortunately, when I tried to start it I received the following error: “Server.InternalError: Internal error on launch”.

I posted a question on the EC2 forum and, as you can see, was told that the root volume was in an “abnormal attachment state”. You’ll also see that my attempts to force a detach via the AWS Management Console appeared to fail or at least, the console thought the volume was still attached and therefore wouldn’t let me reattach it. Reverting to the command line utilities again allowed me to successfully detach and reattach the volume and then start the instance. Another delay whilst the DNS refresh took effect (perhaps I should try out an Elastic IP) and, about three days after I first hit stop, the site is finally back online. I’m putting it down as a learning experience!

Server hacked

Wednesday, March 9th, 2011

When attempting to post the image for my last blog entry, it failed to be resized. When I logged in to the server to see what was up with ImageMagick, I didn’t appear to have permission to execute ls. Or ps. Or netstat… A quick Google suggested that these were the hallmarks of a rootkit attack. Unfortunately the files had been modified prior to the oldest Slicehost backup that I had. At this point I realised the server was still running Intrepid, limiting my chances of picking up packages to detect and remove rootkits. After a reboot of the server I discovered that I had lost all connectivity. Booting up a Slicehost rescue image I was able to retrieve all of the data I needed. Now to get things up and running again. Earlier in the year I had been playing around with a free micro-instance on EC2 and this seemed like the ideal opportunity to switch across. The instance is running Apache rather than nginx as on my Slicehost image. This needed a bit of tuning down to prevent segmentation faults. Everything seems to be running smoothly now. I just need to switch the DNS records away from Slicehost and then I’m done.

Street View Fame

Monday, March 22nd, 2010

Back in October ’98 I blogged about the arrival of the Google Street View car in our road. The data for our area has finally been made available and I’m glad to report that Emma (with short hair) and I can now claim our moment of fame. I’m disappointed that we don’t get to feature twice despite crossing the road whilst the car turned at the end of the cul-de-sac. Having moved house since you’re not likely to find us in the same spot again!

Entries close

Friday, February 20th, 2009

Entries for the JOK Chasing Sprint closed this evening which was a great relief. This was to have been the year of the Web 2.0 entry system but, in reality, other commitments meant that my annual rewrite of the system only got as far as a stateless interface. This should stand me in good stead for next year though… Rails still continues to be a pleasure to use and the more I learn (I splashed out on the PDF version of the soon to be released Agile Web Development with Rails: Second Edition) the better it gets. New improvements to the development process this year were Git for version control and Capistrano for deployment. These certainly gave me more a lot more confidence in making updates to the live system.

(more…)

WordPress upgrade

Friday, January 9th, 2009

I’ve finally had a chance to upgrade to WordPress 2.7. It’s relatively painless since I’ve switched to updating via Subversion. The only problem I hit was with text widgets. I was seeing an error along the lines of “Warning: array_keys() [function.array-keys]: The first argument should be an array in wp-includes/widgets.php on line 1044” which, if you Google it, currently appears at the top of a whole host of other people’s blogs. Deleting all (three) of my text widgets and recreating them seemed to fix the problem. It’s good to see that threaded comments have finally made it in to the base product. I’ve previously experimented with a couple of plugins but never turned them live for users as I’ve never been entirely happy with the way they’ve been implemented. (As a consequence though you’ll see that some historical posts already have threaded responses from me.) I have to say that, right now I’m not a big fan of the revamped admin pages but that may be just lack of familiarity…