Archive for the ‘Work’ Category

« Older Entries

Helm: for better or worse?

Monday, June 9th, 2025

A few weeks ago, one of my colleagues at JUXT gave a presentation on Helm, and this started me thinking back over my own experiences with the tool. It appears I already had a lot to say on the subject back in 2018! Since then, I’ve made extensive use of Helm at CloudBees where we had an umbrella chart to deploy the entire SaaS platform, and at R3. It’s that latter experience that I’m going to talk about in this post.

Helm and Corda

The main Helm chart in question is the one for the R3’s Corda DLT, which you can find on GitHub. The corda.net website has, unfortunately, been sunset, but my blog post describing the rationale for using Helm is still available on the Internet Archive. Another article explains how the chart can be used, along with those for Kafka and Postgres, to spin up a complete Corda deployment quickly.

As an aside, it was a conscious decision not to provide a chart that packaged Corda along with those Kafka and PostgreSQL prereqs. The concern was that customers would take this and deploy it to production without thinking about what a production deployment of Kafka or Postgres entails. Not to mention wanting to make it clear that these were not components that we, as a company, were providing support for.

As a cautionary tale: despite its name, the corda-dev-prereqs chart referenced in that last article (which creates a decidedly non-HA deployment of Kafka and PostgreSQL) found itself being deployed in places it shouldn’t have been…

More Go than YAML

Whilst the consumer experience with the Helm chart was pretty good, things weren’t so rosy on the authoring side. The combined novelty of Kubernetes configuration and Go templating was just too much for many developers. While some did engage, ownership of the chart definitely remained with the DevOps team that authored the initial version, rather than the application developers.

The complexity of the chart also ramped up rapidly. With multiple services requiring almost identical configuration, we soon moved from YAML with embedded Go to Go with embedded YAML! That problem is not unique to Helm; I remember having the same issue with JSPs many moons ago.

The lack of typing, combined with the fact that all functions return strings, started to make the chart fragile, particularly without any good testing of the output with different override values.

Two charts are not better than one

If you look at the GitHub repository, you might wonder why most of the logic for the chart sits in a separate library chart (corda-lib) on which the main corda chart depends. What you can’t see is that we had a separate Helm chart for use by paying customers. This was largely identical to the open-source chart, but included some additional configuration overrides. The library chart was an attempt to share as much logic as possible between the two.

What we couldn’t share was the values.yaml itself and the corresponding JSON schema, and as a consequence, there was always a certain amount of double fixing that went on. What we really needed was a first-class mechanism for extending a chart.

Helm hooks

Although there were other niggles, the last issue I’m going to talk about is the use of Helm hooks. Corda has two mechanisms for bootstrapping PostgreSQL and Kafka: an administrator can use the CLI to generate the required SQL and topic definitions, or the chart can automatically perform the setup when the chart is installed. We expected customers to use the former mechanism, at least in production, but the latter was used in most of our development and testing, and by the services team in pilot projects. The automated approach used a pre-install hook to drive a containerised version of the CLI to perform the setup.

So far, so good. We then started to look at using ArgoCD to deploy the chart. ArgoCD doesn’t install Helm charts directly, instead, it renders the template and then applies the Kubernetes configuration. It does have some understanding of Helm hooks, converting them into ArgoCD waves, but it doesn’t distinguish between install and upgrade hooks. This would lead ArgoCD to try to rerun the setup during an upgrade.

Now, here some responsibility must lie with the Corda team, as those setup commands should have been idempotent, but they weren’t. The answer, for us, was to use an alternative to ArgoCD (worth a separate post), but our customers might not have the luxury of that choice.

Summary

Does all of the above mean that I think Helm is a bad choice? As always, it depends. For ‘packaged’ Kubernetes configuration, I still believe it’s a better choice than requiring consumers to understand your YAML sufficiently to be able to apply suitable modifications with Kustomize. In particular, pushing Kustomize is opening up your support organisation to having to deal with customers basically using any arbitrary YAML to deploy your solution.

In the case of Corda, we underinvested in building the skills to make the best of Helm. Fundamentally, though, I’d suggest that we simply outgrew it. If I were still working on its evolution, the next step would undoubtedly have been to implement an operator and write all of that complicated logic in a language that properly supports testing and reuse.

Posted in Kubernetes, Work | No Comments »

Lockdown 2

Saturday, February 27th, 2021

October started with Duncan’s birthday. It was a pretty quiet affair at home with Christine’s parents visiting and a film in the evening. Friday night has become film night more generally now (after a trial of Sky Cinema we signed up for a discounted subscription). Trying to select a film is always a bit of a trial, particularly balancing Emma’s desire for action against age-appropriate content for Duncan! I’m usually the one who ends up making a final decision to try and satisfy everyone’s needs.

Swimming lessons continued, as did long walks and cycle rides at the weekend. In addition to Tuesday Night Runs (now with beer inside but with face masks and table service), I managed a couple of runs with the IBM gang. We went to another SCJS training with Duncan, this time in the New Forest. We were back in the Forest a couple of weeks later for an event I planned at Anderwood. I managed to put one control in the wrong place – the perils of doing all of my planning in one visit to the area the week before the event – but otherwise everything went well!

Still with vacation to burn, I had half term off. Much of it was taken up redecorating Emma’s room: out with the little girl pink, in with the teenage grey (there was much debate about quite how dark a grey she could have!). Duncan and I amused ourselves soldering and assembling the MERG DCC command station and handset kits for his railway. Two pairs of hands definitely made the job easier, not to mention Duncan’s eyesight when it came to the surface-mounted components in the handset. Remarkably, it all worked once assembled but the train and track now need a bit of an upgrade to get a reliable signal.

Lockdown returned for four weeks in November which brought the more sociable activities to an end (at least face-to-face). You were still allowed to exercise with one other person, so Tuesday Night Runs involved Ian and I running around Chandler’s Ford, and I could still go out cycling with Alasdair. Through an article in CompassSport, we had discovered the Turf app (think Pokémon GO but without the cute characters) which, even four months later, is still getting us out the door.

Duncan bought himself a small drone which we then promptly had to retrieve from the garden two doors down the road! He also bought himself a penknife which has, so far, only caused the loss of blood on one occasion! The month ended with Christine’s birthday. Come December, Christine was also allowed to tell the world that she had succeeded in her promotion to Professor. At least she now needn’t worry about being called upon to help in an emergency!

Emma’s inflatable boot had not helped her ankle but we were then left waiting for the hospital to start operating on children again. We were eventually given a date at the start of December (still over a month before the NHS would have even started to look at her ankle). Christine would go with her so both had to traipse up to Basingstoke for a Covid test three days beforehand, and then the whole family had to isolate until the day itself. Having been on the front of the list, Emma was awake again by mid-morning and home early afternoon. All seemed to go well and, rather than a plaster cast, she got to use the inflatable boot again. By Christmas, she was walking around normally again although is still a bit wary of cycling.

Things opened up again for the next few weeks. Orienteering resumed with events at Bramshaw and Farley Mount. We even managed the annual Run the Pubs, albeit that the meal afterwards had to be in the pub garden as we were not all from the same household. I managed to give blood (having been turned away with a sore throat on the previous occasion). We also had a big online launch event for the Software Delivery Management product I’m working on, for which everyone in the company received a set of glassware in the post so we could drink a toast. (You had to provide your own drink though!)

The dreaded R-number was on the rise again in the run-up to Christmas and there was much debate about what we might be able to do. At best, it was possible that we might be able to meet family for a few hours outside but, by the 19th, Hertfordshire (where my family is based) was put in the topmost tier, ruling even that out. By Boxing Day, we joined them in Tier 4 and we were to enter another national lockdown in the New Year. The Christmas period was therefore spent at home, enjoying the frosty conditions outside, and doing yet more decorating!

New Year’s Eve was made a bit special as we cashed in the money that CloudBees was contributing to a festive meal and had a nice takeaway. Emma was then determined to stay up until midnight and we thought we should probably keep her company! As the year came to an end, I’m sure everyone had the same wish: that 2021 should be a better one.

Posted in Family, Food, Friends, Orienteering, Running, Work | 1 Comment »

And we’re back!

Wednesday, February 10th, 2021

From time-to-time, I feel the urge to write a blog post on something and then I realise that there’s this gaping void since I last wrote anything (16 months) for which it would be remiss not to say anything given all that has happened. So, the next few posts will be a whistle-stop tour of that gap, hopefully, followed by some more regular posting. The advantage of going back over a year is that I get to relive life pre the big ‘C’ (that’s Covid, not Cancer, for those looking back at some point in the future when hopefully this is all but a distant memory).

So what happened between Duncan’s birthday and the end of 2019? There was lots of outdoor activity, starting with Christine running the Clarendon Marathon. I did a couple of the CC6 cross-country events, as well as two Hants XC League races. The one at Sparsholt college was a particularly good mudbath! I was also continuing to run the monthly Strava challenges set by my ex-colleagues at IBM and returning to Hursley every other Wednesday to either run or marshall the fit52 5k events.

On the orienteering front, I planned an event at Fritham which was well attended despite the atrocious weather. After hanging controls, the rest of the family disappeared to get the car MOT’d as we’d discovered the night before that it had expired! I managed to leave one of the controls out in the forest but thankfully was back there a couple of days later for a Tuesday evening run and could retrieve it.

During half-term, we all went along to a Military League South event at Roundhill. It was the November Classic at the end of that week and I had the fun job of parking cars in an area that we were sharing with Totton RC’s Stinger race. It seemed to work okay although I then had a terrible run on Bramshaw. Emma was meant to attend an O-Camp at Burley Youth Hostel that weekend but it ended up being only a single night due to more bad weather.

We also managed another club event at Kings Garn Gutter and the British Schools Orienteering Champs near Slough where both children had good runs. SOC was also having monthly MapRun events taking us to Winchester, Romsey, and Southampton.

Music was another theme with Christine performing in two Thornden Community Wind Band concerts, Duncan taking his Grade 3 recorder exam, and Emma appeared in a school concert singing and playing the violin and recorder (not all at once). Not sure what my contribution to this them was!

On the work front, I had two overseas trips. An internal meeting in Raleigh (again) and then I was on booth duty at our DevOps World event in Lisbon (a first visit to Portugal for me). I was interviewed for, and offered, a job which would have seen me return to an office location. After much soul searching and discussion with my current management, I decided to turn the offer down but it did precipitate my move into a tech lead role at CloudBees.

Indirectly related to work, I gave one of Christine’s lectures on “Databases and SQL”, a subject that has been occupying a disproportionate amount of my time. I also helped out at a Code Retreat back at IBM which was good fun. I made a last-minute decision to go to the London Java Community’s Unconference. I almost didn’t make it when my key got stuck in my bike lock at the station but, with some WD40 courtesy of SW Trains, I was on my way again. There were some good sessions as well as providing an opportunity to catch up with some old friends.

Despite many an hour spent completing Advent of Code (in Python this year), there was still time for some socialising in the run-up to Christmas starting with the CloudBees Whitely Christmas meal. The “Run the Pubs” tradition continued, albeit with less running and fewer pubs! It was also our turn to host a group of friends for pre-Christmas drinks. We’d just about recovered from that before disappearing to Monmouth for Christmas itself. We then made a trip across to my parents before New Year which was to prove to be the last time we’d see them face-to-face for over a year…

Posted in Family, Orienteering, Running, Travel, Work | Comments Off on And we’re back!

Knative Intro @ Devoxx UK

Thursday, May 30th, 2019

I presented an introduction to Knative at Devoxx UK, the recording for which can be found below. I’m afraid I deviated somewhat from the abstract given the changes to the project in the five months since I submitted it. With only half an hour, I probably shouldn’t have tried to cover Tekton as well but I wanted to have an excuse to at least touch on Jenkins X, however briefly! The demo gods largely favoured me except when hey failed to return (not the part of the demo I was expecting to fail!). The script and source for the demo are on GitHub although I’m afraid I haven’t attempted to abstract them away from the Docker Hub/GCP accounts.

Posted in Kubernetes, Work | Comments Off on Knative Intro @ Devoxx UK

Debugging with Telepresence

Monday, February 11th, 2019

I’ve spent the last few days trying to debug an issue on Kubernetes with an external plugin that I’ve been writing in Go for Prow. Prow’s hook component is forwarding on a GitHub webhook and the plugin mounts in various pieces of configuration from the cluster (the Prow config, GitHub OAuth token and the webhook HMAC secret). As a consequence, running the plugin standalone in my dev environment is tricky, but just the sort of scenario that Telepresence is designed for.

The following command is all that is needed to perform a whole host of magic:

  • It locates the my-plugin-deployment deployment already running in the cluster and scales down the number of replicas to zero.
  • It executes the my-plugin binary locally and creates a replacement deployment in the cluster that routes traffic to the local process on the exposed port.
  • It finds the volumes defined in the deployment and syncs their contents to /tmp/tp using the mount paths also specified in the deployment.
  • Although not needed in this scenario, it also sets up the normal Kubernetes environment variables around the process and routes network traffic back to the cluster.

Now, it was convenient in this case that the binary already exposed command line arguments for the configuration files so that I could direct them to the alternative path. Failing that, you could always use Telepresence in its--docker-run mode and then mount the files onto the container at the expected location.

And the issue I was trying to debug? I had used the refresh plugin as my starting point and this comment turned out to be very misleading. The call to configAgent.Start() does actually set the logrus log level based on the prow configuration (to info by default). As a consequence, everything was actually working as it should and my debug statements just weren’t outputting anything!

Posted in Kubernetes, Work | Comments Off on Debugging with Telepresence

Farewell IBM

Thursday, August 2nd, 2018

On 2nd August, I handed back my IBM badge, just shy of twenty years after I first joined the company. I’ll come back to the ‘why?’ and ‘where next?’ questions and start with a recap of those intervening years (with apologies for the consequent length of this post!).

I started at IBM Hursley on 6th October 1998, fresh out of university with a degree in Engineering and Computer Science. I was a month late for the beginning of the graduate programme having taken some time out to travel across Canada by Greyhound coach! I began working on IBM’s C++ CORBA offering (Component Broker) with a brief spell in test before switching to development in the transactions team. (Remember when ‘test’ and ‘development’ were two different teams?) Many of my colleagues in that team (too many to name but they should know who they are) formed the basis of a network that would define the shape of my future career. (My Component Broker mug is still going strong but I’m afraid I ditched the set of foils describing the product that I found when clearing out my desk!)

At university I’d used, the then nascent, Java in a couple of projects and those skills were to become of use as we started to add a Java client. Before long, the focus switched to the newly-defined J2EE specifications and WebSphere Application Server was born. After working on the JTA and Activity Session implementations, I joined a team looking at integration with MQ. When the time came to implement an embedded JMS provider in WebSphere Application Server V6, it was natural I should move to work on that.

Six years in, I was starting to make architectural decisions but desired a better understanding of how customers were actually using our products. When the opportunity came up to work as a software consultant in IBM Software Services for WebSphere (aka Lab Services), I jumped at the chance. The next few years were spent travelling across Europe, doing everything from performance bake-offs, resolving critical situations, to participating in first-of-a-kind projects. I particularly enjoyed this time, learning to survive on your wits on those occasions when it wasn’t possible to draw on that all important network. This was also the period during which this blog began.

On returning from a short-term assignment to Norway a, by now one-year old, daughter meant it was time to get my feet back under a development desk. Having worked with customers on WebSphere ESB, it was natural to join that team. From there, I had the pleasure of building and leading a new development team to take over what was to become WebSphere Appliance Management Center. We had great fun, rewriting the offering to build on the new WebSphere Liberty Profile with a shiny new JavaScript front-end (thankfully IBM later moved on from Dojo though) in what I still think was one of the most passable efforts at agile I’ve seen in IBM.

Eventually, the team were moved to work on IBM API Management. The eight-hour time zone difference to the half of the team in California didn’t work for me and, after a nine-year break, I rejoined the WebSphere Application Server family. Initially, I was working on the open source Cloud Foundry buildpack. A side project relating to Netflix OSS was the start of an interest in microservices. From there, I lead efforts relating to containerization, including the publication of official images on Docker Hub.

This, in turn, led to Microservice Builder: a platform for developing, building and deploying, cloud-native applications on Kubernetes. This was then rolled into an offering called Microclimate which added a greater emphasis on the developer experience and that brings us to the current day.

So why, after so many years working with such great people on such a variety of interesting projects, am I now set to leave? Sure, there have been frustrations in working for IBM, but I’m sure many of those are common to all large, shareholder-owned, multi-national companies. As an example, take the laying down of corporate instructions that mandate that all 380,000 employees be treated in some particular way that cannot possibly be equally applicable to all. Thankfully I’ve been blessed with managers who have all excelled in the flexible interpretation of those rules. Many of those same managers are helping to revive Hursley as the vibrant technical community that I first joined.

Really, my departure just boils down to wanting to experience working for a different company. I’ve often said that IBM is the best employer within a two-mile radius of my house and I’ve set a lot of store by that convenience. My LinkedIn profile has been ‘open to offers’ for a few years now but I’ve been resistant to the lure of London money/startups or the peripatetic life of the solution architect. In this case though, I was offered the opportunity to work from home, not as the lone outcast, but for a company that is almost entirely distributed. It was also an opportunity that would utilize the skills around the cloud and DevOps (in particular Kubernetes and Jenkins) that I’ve garnered over the past few years. Such is the overlap that I even credited one of my technical interviewers in a presentation I gave earlier this year when citing their work!

So, without further ado, from 28 August I will be a Senior Sofware Engineer at CloudBees where I’ll be joining the architecture team for their core (Jenkins) offering. At eight-years-old, the company is very much a late-stage venture but, with the distribution list for my leaving email at IBM having more people on it than there are in the entire company, it will be quite a different prospect to working at IBM. Much more than that, I can’t tell you because, quite frankly, I don’t know, but I’m looking forward to new colleagues and challenges. Stay tuned to this blog to find out what happens next!

 

Posted in Work | 2 Comments »

Scratching at Work

Saturday, May 12th, 2018

Not satisfied with a four-day Bank Holiday week, I was back in work today for a Scratch Day organised by the inimitable Dale Lane, supported by an all-star cast of IBMers, past and present. The day got off to an ‘exciting’ start with Duncan and I cycling there along Hursley Road. Emma joined us by car, just as the day got going, hot foot from her swimming lesson.

There was a good turnout from IBM and other local families. On offer was a selection of projects from Code Club and Dale’s own Machine Learning for Kids. Emma and Duncan worked separately and I probably spent most of my time helping Duncan (although both are familiar with Scratch from school and home). Typically, Duncan picked two of the ‘advanced’ options but, having heard Dale talk about them at a lunchtime session, I was more than happy to try out a couple of the ML exercises.

We started with Judge a book which performs image classification on book covers to try and identify genre. I was a bit slow to realise that Duncan was logged in to my Amazon account whilst performing his searches but thankfully we switched to an incognito session before getting to the flesh-covered books under Romance! He’d picked Horror and Fantasy as two of his other genres and it wasn’t surprising that the classifier occasionally got those confused.

I had to help out a fair amount with the Headlines exercise as there was a lot of typing to enter the training set from different newspapers. We didn’t manage to finish before the end of the day but we still had an interesting discussion about the differences between tabloid and broadsheet headlines.

The event closed with an opportunity for the children to show what they had done to the others. Although some were a little reticent, this was a great opportunity for them to build a little confidence and soak up the applause that each invariably got.

All-in-all, we had a great day and my thanks go to all those that gave up a day (and more) to help out. We’ll certainly be checking out a few of the other projects and hope that Scratch Day makes a return to Hursley next year.

Posted in Family, Technology, Work | Comments Off on Scratching at Work

Index Developer Conference

Sunday, February 25th, 2018

IBM launched a new conference in San Francisco under the name Index and I was lucky enough to attend. This wasn’t your usual IBM conference focused on brands and products. Although the tracks were aligned with IBM’s strategic areas (Cloud, Blockchain and AI talks were much in evidence, for example) it really was a developer conference with keynotes and speakers from well-renowned figures across the industry.

You can watch my session covering deploying Jenkins on Kubernetes with Helm and deploying to Kubernetes from Jenkins with Helm below. You can find the deck on SlideShare and the demo material on GitHub. For those who know what I work on, it will be no surprise that this is based on our discoveries when developing Microservice Builder. I highly recommend you also check out some of the other sessions on the conference playlist and watch out for Index 2019!

https://youtu.be/xzbMHj1ly9c

The timing of the conference meant I had Friday to be a tourist with some colleagues. We headed over to SF MoMA and then made the most of the sunshine with a stroll along the waterfront to see the sea lions and then to have to have lunch overlooking the bay.

Posted in Kubernetes, Travel, Work | Comments Off on Index Developer Conference

« Older Entries