Archive for the ‘Technology’ Category

Kustomizing Kubernetes Konfiguration

Thursday, October 11th, 2018

Finally, I get to write that blog post on kustomize! kustomize is yet another tool attempting to solve the problem of how to make Kubernetes configuration re-usable. Unlike, say, Helm, kustomize allows configuration to be overridden at consumption time without necessarily having allowed for it when the configuration was originally produced. This is great if you are attempting to re-use someone else’s configuration. On the flip-side, you might prefer to use something like Helm if you actually want to limit the points of variability e.g. to ensure standardization across environments or applications.

You know the drill by now: the go binary CLI can be obtained via brew install kustomize. There is one main command and that is kustomize build. That expects to be pointed at a directory or URL containing a kustomization.yaml file. Running the command outputs the required Kubernetes resources to standard output where they can then be piped to kubectl if desired.

The kustomization.yaml can contain the following directives:

  • namespace – to add a namespace to all the output resources
  • namePrefix – to add a prefix to all the resource names
  • commonLabels – to add a set of labels to all resources (and selectors)
  • commonAnnotations – to add a set of annotations to all resources
  • resources – an explicit list of YAML files to be customized
  • configMapGenerator – to construct ConfigMaps on the fly
  • secretGenerator – to construct Secrets via arbitrary commands
  • patches – YAML files containing partial resource definitions to be overlayed on resources with matching names
  • patchesJson6902 – applies a JSON patch that can add or remove values
  • crds – lists YAML files defining CRDs (so that, if their names are updated, resources using them are also updated)
  • vars – used to define variables that reference resource/files for replacement in places that kustomize doesn’t handle automatically
  • imageTags – updates the tag for images matching a given name

That’s a pretty comprehensive toolbox for manipulating configuration. The only directive I didn’t mention was bases with which you can build a hierarchy of customizations. The prototypical example given is of a base configuration with different customizations for each deployment environment. Note that you can have multiple bases, so aws-east-staging might extend both aws-east and staging.

One of the refreshing things about kustomize is that it explicitly calls out a set of features that it doesn’t intend to implement. This introduces the only other command that the CLI supports: kustomize edit. Given that one of the stated restrictions is that kustomize does not provide any mechanism for parameterising individual builds, the intent of this command is to allow you to script modifications to your kustomization.yaml prior to calling build.

It’s worth noting that kustomize can be used in combination with Helm. For example, you could run helm template and then use kustomize to make additional modifications that are not supported by the original chart. You can also use them in the reverse order. The Helmfile docs describe how to use Helmfile’s hooks to drive a script that will use kustomize to construct the required YAML, but then wrap it in a shell chart so that you get the benefit of Helm’s releases.

Helmfile and friends

Monday, October 8th, 2018

Having written a post on Helm, I feel obliged to follow it up with one on Helmfile, a project that addresses some of the issues that I identified with deploying multiple Helm charts. In particular, it provides an alternative approach to the umbrella chart mechanism that Jenkins X uses for deploying the set of charts that represent an environment.

Yet again, we have a go binary, available via brew install helmfile. At its most basic, we then have a helmfile.yaml that specifies a set of releases with a name, chart, namespace and override values for each. A helmfile sync will then perform an install/upgrade for all the releases defined in the file. One thing I failed to mention in my Helm post was that Helm supports plugins on the client side. One such plugin is the helm-diff plugin which, as you’d probably guess from the name, gives you a diff between the current state of a release and what it would look like after an upgrade or rollback. The plugin is installed with:

With this in place, we can now use helmfile diff to see the changes that would take place across all of our releases. The helmfile apply command combines this with a sync to conditionally perform an upgrade only if there are differences. There is a set of other helmfile commands that all perform aggregate operations across all the releases: delete, template, lint, status and test.

So far so good but nothing that couldn’t be achieved with a pretty short bash script. Where things get more interesting is that the helmfile.yaml is actually a template in the same way as the templates in a Helm chart. This means we can start to do more interesting things like defining values in one place and then reusing them across multiple releases. Helmfile has the explicit concept of an environment, passed in as a parameter on the CLI. We can use a single YAML file and use templating to have different values apply in each environment or, in the extreme, only deploy charts in some environments.

Helmfile also has some tricks up its sleeve when it comes to secrets. Most trivially, if your CI allows you to configure secrets via environment variables you can consume these directly in the helmfile.yaml. You can also store secrets in version control encrypted in a YAML file and then have another Helm plugin, helm-secrets, decrypt them with PGP or AWS KMS.

Helmfile has some features to help you as the size of your deployment grows. You can, for example, specify a selector on commands to only apply them to matching releases. This can be helpful if deploying all the changes at once is likely to create too much churn in your cluster. You can also split the file into multiple files in one directory (executed in lexical order) or over multiple directories (accessed via a glob syntax).

For anything else, there are prepare and cleanup hooks to allow you to execute arbitrary commands before and after deployment. Oh, and if you’re using a containerized deployment pipeline, it’s available packaged up in an image, ready for use. Finally, if you don’t take to Helmfile, take a look at Helmsman instead.

Helm: the Package Manager for Kubernetes

Friday, October 5th, 2018

I wanted to take a look at ksonnet and kustomize  but felt I should write about what I know best first, and that’s Helm. All three tools are trying to tackle the same basic problem: enabling the re-use of Kubernetes configuration where typically we want some level of customisation each time we use the configuration, whether that’s to reflect the latest deployment or the environment we’re deploying to.

The starting point for Helm is a client binary called, unsurprisingly, helm, and is available from brew as kubernetes-helm. The current version of Helm also has a server-side component called Tiller and that is deployed by executing helm init --wait (the wait flag indicates to wait until the Tiller pod has started before returning). Note that Helm is pretty picky about having matching versions of client and server.

In keeping with Kubernetes nautical theme, Helm’s unit of packaging is the chart. A new chart called test can be easily constructed with the command helm create test. This results in a directory called test with the following contents:

Chart.yaml contains some basic meta-data about the chart such as name, description and version. Note that Helm draws a distinction between the version of the chart and the version of the application that the chart deploys. We’ll come back to the empty charts directory later. The templates directory contains the YAML files that we’d expect to find for a basic Kubernetes application and these can be extended and/or replaced as needed. As the directory name suggests though, and this is the key to reuse, the files are actually templates. If we look in deployment.yaml we’ll see the mustache syntax used to support substitution:

We can see that it’s going to use the name from the Chart.yaml for the container name. The default contents for the Values fields are taken from values.yaml and typically this also contains comments describing the intent of each value. The templates use Go template syntax and support the sprig functions along with a few Helm-specific extensions. The syntax is pretty rich and supports pretty much any manipulation you’re likely to want to perform on configuration. The _helpers.tpl file defines some helper functions that are used throughout the chart. Finally, the NOTES.txt contains text that is output when the chart is installed, typically providing usage instructions for the installed application. This file also supports templating.

The default generated chart deploys a pod with nginx in it. If we look in values.yaml we can see that, by default, it’s going to use the stable image tag. We can override this at install time. For example:

If we want to override lots of values then we can specify then via a YAML file instead. Indeed, we can specify multiple files so there might, for example, be one for production and another for us-east-1. The name here is optional but, at least for scripting, it’s easier not to use one of the generated names. Note that, although the release is deployed to a namespace (and Helm isn’t capable of tracking resources that explicitly target some other namespace), the name is scoped to the tiller instance (i.e. the cluster if you only have one tiller install).

There are others commands for operating on releases: delete, upgrade, rollback, list, status and get, all of which do pretty much what you’d expect of them. As the upgrade, rollback and history commands suggest, helm is tracking revisions of a release. Tip: if you’re in some CD pipeline and you don’t know whether you’re upgrading or installing for the first time, use helm upgrade --install and it will do the right thing. Something else to watch out for is the --wait option on an upgrade. This waits until you have at least replicas - maxUnavailable pods available. Given that these are both typically one by default, don’t be surprised when it doesn’t appear to be waiting!

We’ve just installed a chart off the local filesystem but, as with any decent package manager, Helm has the concept of a repository. By default, the helm CLI is configured with the stable and incubator repositories. You can search these with helm search or head over to Kubeapps for a shiny catalog. These show the real power of Helm when, with a simple helm install stable/wordpress, you can re-use someone else’s hard work in defining the best practise for deploying WordPress on Kubernetes. You can add other repositories (e.g. that for Jenkins X) and you can create your own, either via a simple file server or, a read-write repository using ChartMuseum and the Monocular UI.

The packages themselves are simply .tgz files although the helm package command also supports things like signing the package. I said I’d come back to the charts directory and the WordPress chart is a good example of how this can be used. The WordPress chart actually also deploys the MariaDB chart to provide persistent storage. This is achieved by placing a requirements.yaml in the root of the chart that specifies the dependency. Dependencies can be fetched with helm dependency update or resolved at packaging time with the --dependency-update flag.

Jenkins X makes use of this sub-chart capability in the way it deploys environments. The repo for each environment contains a Helm chart which specifies as dependencies all of the application charts to deploy in to the environment. One downside of this approach is that you then only see a Helm release for the entire environment. Another issue with sub-charts relates to the way their values are overridden. The values for a sub-chart can be overridden by prefixing with the name or alias given in requirements.yaml but there is no good way to get the same value in to multiple sub-charts. If you have control over the sub-charts you can write them to retrieve values from a global scope but that doesn’t help if you’re trying to re-use someone else’s efforts.

Helm provides lots of other goodness. You can annotate resources with hooks so that they are run pre or post install, upgrade, rollback or delete. There is even a crd-install annotation to ensure that your CRDs are created before other resources attempt to use them. You should also know helm lint and helm test. The latter executes resources in the chart with the test-success or test-failure hook annotations. You can use these to provide a detailed check of whether a release was successfully deployed.

An overview of Helm wouldn’t be complete without a reference to Helm v3. At the beginning I indicated that the current version deploys a server-side component. That is particularly problematic when it comes to security as, even if you enable mutual-TLS, any user that can connect can perform any action supported by the service account under which Tiller is running, losing you all of Kubernetes’ RBAC benefits. You can mitigate the problem by installing a Tiller per namespace but you’re still left managing certificates. IBM Cloud Private implemented their own Tiller to do identity assertion in order to overcome this problem. Jenkins X now supports using helm template to render charts client-side before deploying with kubectl. This means that you don’t get any of the Helm release management but then that is handled by Jenkins X anyway. Helm 3 promises to do away with Tiller but still share release information server-side via CRDs. Sadly as James alludes to in his blog post, there’s not a lot of public progress being made on Helm 3 at the moment.

Continuous Development with Skaffold

Thursday, October 4th, 2018

Next on the list of projects utilised by Jenkins X (and this is a theme that could run and run) is Google’s Skaffold. There is an intersection between the capabilities of Skaffold and Draft. Skaffold does not provide anything like packs (this is where Jenkins X uses Draft) so you need to provide your own Dockerfile and Kubernetes configuration (raw YAML, kustomize templates, and Helm charts are supported). Both projects, however, aim to simplify the process of building an image and deploying it to a Kubernetes cluster to speed iterative development of applications that have hard dependencies on a Kubernetes environment, or the services running therein.

In the Skaffold case, a skaffold run will do a one-time build and deploy and a skaffold dev will continuously monitor the filesystem to determine when to rebuild and update. As previously discussed, the value of being able to do this versus needing a smart incremental deployment from an IDE is very much dependent on how quick that rebuild process is for your application language/runtime of choice. As with Draft, it has the ability to allow you to skip pushing to a registry when you’re working with a local cluster.

So what does Skaffold offer that Draft doesn’t? Principally, that it is not designed to be used solely at development time. The idea is that the same skaffold run may also be used as part of your continuous deployment pipeline. If you’re a GCP user, this extends to capabilities like using Google Cloud Builder or Kaniko rather than a simple Docker build and, of course, interaction with a registry.

As this annotated skaffold.yaml shows, it has a few other neat tricks. You have lots of flexibility over the tagging scheme used for images: SHA, git commit ID, timestamp, or a Golang template. For a Docker build, you can specify build arguments and cache images. You can even configure a bazel build instead of a Docker build.

One thing to watch is that you adhere to the convention used for substituting the names of the built images. When using Helm, for example, the default behaviour is to pass the combination of image repository/name and tag as a single value. If you chart is using the default Helm convention of separate .repository and .tag values then you need to specify a different imageStrategy. If your chart expects a three-way split between repository, name, and tag, then you’re on your own!

Developing in-cluster with ksync

Sunday, September 30th, 2018

Continuing on the theme of technologies used by Jenkins X under the covers, this post is going to take a look at ksync. In Jenkins X, this project is used to implement the ‘DevPods‘ capability. It’s a pretty thin veneer over the top of what is provided by ksync though. As the name suggests, the project performs synchronisation, in particular, bi-directional synchronisation between a directory on your laptop and a pod in a Kubernetes cluster. The aim is to allow you to code locally in your favourite editor but run that code in-cluster where you have access to a full Kubernetes environment, including any other services you might be running.

The workflow is simple: a <code>ksync init sets up your local environment and deploys a daemon set in the cluster to get access to the file system of containers. Next, a ksync watch runs a process locally to monitor your laptop. The last step is to issue a ksync create to form a mapping between a local directory and a directory in a container (or potentially multiple containers that match the selector). And that’s all there is to it: make a change locally and it’s reflected in the container; see files written by the container appear on your local disk.

ksync is, in turn, using syncthing to perform the actual file synchronisation. In addition, it provides the option to restart the application container following a sync (for example, if the application process is not dynamically reloading modified files).

In practise, it works well and certainly supports a much more iterative development experience than you would get if you relied upon a Docker build from something like <code>draft up</code>, even assuming you had optimised your Docker build so it just needed to copy in the application files. As when running an application locally, all of this works best for interpreted languages like PHP, Python or Node. For a language like Java, you need to get the compiled application (or classes) in to the synced directory and that is unlikely to give you the same experience that you can get using an IDE capable of hot-reloading. That’s something that Microclimate looked to address…


draft up

Friday, September 21st, 2018

For my next few posts, I thought I’d pick on some of the technologies that Jenkins X uses under the covers. The first of these is Draft, originally from Deis but it went with them to Microsoft. Draft’s aim is to streamline the process of developing code that runs on Kubernetes. It’s evolved a bit since it was originally released: having started with a client-server architecture, it is now entirely client based. There are many good reasons for this although one of the things that differentiated Draft originally was that it didn’t need anything on the developer’s machine other than Draft itself: not even Docker.

The part of Draft that Jenkins X uses is the ability to add a Dockerfile and Helm chart to an existing project. The combination of Dockerfile and Helm chart is stored in what Draft calls a ‘pack’. On running a draft create, Draft does some nifty analysis to detect the language being used in the project in order to select the appropriate pack to use. As you and I know though, language alone is not going to tell me whether I’m, say, running an executable JAR or providing a WAR file to run on an app server. Fortunately, there’s a --pack option so I can tell Draft which pack to use. The pack mechanism is nicely extensible with the ability to specify new repositories (simply a Git repo containing a packs folder). The packs used by Jenkins X (which include one for Liberty even if it isn’t very good) can be found here. Draft is also clever enough to know that, if I already have a Dockerfile or chart, I probably don’t want the one from the pack instead.

Once I have a Dockerfile and chart, the next step is to deploy my application using draft up. Draft’s expecting to find the Kubernetes context and Helm already set-up. It’ll build the Docker image and if a registry is configured push the image there. The latter isn’t compulsory though so if I’m using Docker Desktop (the new name for Docker for Mac/Windows) or have my Docker client pointing at my Minikube Docker daemon then I can just use the image out of the cache. It will then use Helm to deploy the application, passing through overrides for image.repository and image.tag to reference the image that’s just been built (using a unique tag). It will even set up an imagePullSecret if necessary. You can use draft logs to see the output from the build and deploy.

Originally, Draft came with a ‘watch’ mode where it would attempt to detect file updates and automatically rebuild. Thankfully that now seems to have been dropped as, with a completely unoptimised build cycle, it really wasn’t practical. The Java pack is particularly bad as the provided Dockerfile doesn’t even attempt to cache the Maven dependencies. Now you simply run draft up again to trigger a rebuild (which you could hook up to your editor’s save option if you really wished).

The last part of the Draft developer experience is draft connect which pipes the logs from any deployed containers to your terminal, along with setting up port forwarding. Sensibly, it allows you configure the local ports that you want to forward to and this, along with other configuration, can be stored in a draft.toml file with your application. (The authors have to be congratulated for breaking with the current trend and using TOML rather than YAML!)

There are a few extra niceties in that you can define additional plugins (arbitrary commands that share the Draft meta-data via environment variables) and you can define tasks for a project that execute pre-up, post-deploy, and on cleanup. If, like me, you left wondering where these are documented, check out the Draft Enhancement Proposals where they were introduced.

All-in-all, there is nothing here that couldn’t be achieved here by scripting together a few standard commands but just because it’s simple, doesn’t mean that it isn’t useful. It’s one of many projects that are attempting to reduce developer friction when deploying to Kubernetes, and you can expect a few more posts covering others…

Jenkins X

Wednesday, September 19th, 2018

Having started to get some rhythm back in to the publishing of my personal blog posts, I thought it was about time that I started posting some technical content again too. I’m having to do lots of new learning at the moment and, if nothing else, writing about it makes sure that I’ve understood it and helps me remember at least some of what I’ve learnt. As before, these posts in no way indicate the position of my employer nor, in general, should you read in to them anything about technical direction. On the whole, they are just about topics that I’ve found sufficiently interesting to write a little about. There is, I have found, no knowing what will be of interest to other people (my all-time top post relates to Remote Desktop!). From an entirely selfish perspective, I don’t care if anyone reads what I write as it’s largely the writing that gives me value!

Having said all the above, the subject of this post is Jenkins X which is very much in the domain of my new employer! When it was announced back in March, I have to admit that I was somewhat sceptical. It was clearly aiming at much the same space that we were with the dev-ops part of Microclimate. My view wasn’t helped by the fact that I couldn’t actually get it to run. It did (and still does) run best out on public cloud but I used up my free quotas on AWS and GCP a long time ago. I tried to run it on minikube and failed. It was also developed by the team behind Fabric8 which, although it showed lots of promise, was never incorporated in to any of RedHat’s commercial offerings. The same was not set to be true of Jenkins X and, six months later, my new employer has just announced that it now forms part of the CloudBees Core offering under the name of Kube CD. I’ll save details of that commercial offering for another post and restrict myself to talking about the open source Jenkins X project here.

So what exactly is Jenkins X? It enables Continuous Integration and Continuous Deployment, of applications on Kubernetes. It happens to use Jenkins as the engine to perform those actions but, at least at a first pass, that is immaterial. Around that Jenkins is wrapped lots of Kubernetes-native goodness and, most importantly, a CLI by the name of jx. Thankfully this time around the minikube experience worked for me just fine and getting up and running was as simple as:

I have to say that I’m not a big fan of ‘verb followed by noun’ when it comes to CLI arguments as, although perhaps more readable, it makes end-user discovery harder (jx create tells me about a whole long list of largely unrelated things) but thankfully just typing jx gives a reasonable overview of the main options. Beware though that the CLI is heavily overloaded: it’s not only used for initialisation, but also subsequent actions performed by the developer, and those performed by the pipeline.

Perhaps the quickest way to demonstrate the capabilities is to then use a quick start:

This allows you to select a technology stack (everything from Android to Vert.X via Rust, Rails and React!) which lays down a skeleton application on disk. It then uses Draft to add a Dockerfile and Helm chart(s). It doesn’t stop there though. It will then help you create a repo on GitHub, check your code in, set up a multi-branch pipeline on the Jenkins instance it provisioned, and set up the web hook to trigger Jenkins on subsequent updates. (Webhooks don’t tend to work too well unless your minikube is internet facing but, given a bit more time, polling does the job eventually.) The default pipeline (defined by a Jenkinsfile in your application repository) uses Skaffold to build the application Docker image and push to a registry. The Helm chart is published to the provided instance of ChartMuseum.

Jenkins X follows the GitOps model promulgated by Alexis Richardson and the team at Weaveworks. By default, it sets up two GitHub repositories that map to staging and production namespaces in the Kubernetes cluster. Additional environments can easily be defined via, you guessed it, jx create environment. These repositories make good use of ‘umbrella’ Helm charts to deploy specific versions of each of the application charts. By default, the master branch is automatically deployed to the staging environment but promotion to production is performed manually, for example:

There is also the concept of a preview environment. Typically created for reviewing a pull request, they can also be created manually via the CLI. These allow a specific version of the application to be accessed in a temporary namespace created just for that purpose. All of the Jenkins X configuration (environments, releases, …) are represented in the Kubernetes way: as Custom Resource Definitions.

There’s plenty more to say about Jenkins X but I’ll save that for another post on another day. Hopefully this has given you enough of a flavour to encourage you to download the CLI and give it a try for yourself.

Scratching at Work

Saturday, May 12th, 2018

Not satisfied with a four-day Bank Holiday week, I was back in work today for a Scratch Day organised by the inimitable Dale Lane, supported by an all-star cast of IBMers, past and present. The day got off to an ‘exciting’ start with Duncan and I cycling there along Hursley Road. Emma joined us by car, just as the day got going, hot foot from her swimming lesson.

There was a good turnout from IBM and other local families. On offer was a selection of projects from Code Club and Dale’s own Machine Learning for Kids. Emma and Duncan worked separately and I probably spent most of my time helping Duncan (although both are familiar with Scratch from school and home). Typically, Duncan picked two of the ‘advanced’ options but, having heard Dale talk about them at a lunchtime session, I was more than happy to try out a couple of the ML exercises.

We started with Judge a book which performs image classification on book covers to try and identify genre. I was a bit slow to realise that Duncan was logged in to my Amazon account whilst performing his searches but thankfully we switched to an incognito session before getting to the flesh-covered books under Romance! He’d picked Horror and Fantasy as two of his other genres and it wasn’t surprising that the classifier occasionally got those confused.

I had to help out a fair amount with the Headlines exercise as there was a lot of typing to enter the training set from different newspapers. We didn’t manage to finish before the end of the day but we still had an interesting discussion about the differences between tabloid and broadsheet headlines.

The event closed with an opportunity for the children to show what they had done to the others. Although some were a little reticent, this was a great opportunity for them to build a little confidence and soak up the applause that each invariably got.

All-in-all, we had a great day and my thanks go to all those that gave up a day (and more) to help out. We’ll certainly be checking out a few of the other projects and hope that Scratch Day makes a return to Hursley next year.