David Currie

A short post for today but it relates to a tool that every Kubernetes user should have in their toolbox: kail. Although most users probably know that kubectl logs will, by default, show the logs for all containers in a pod and that it has --tail and -f options, fewer probably know that is has a -l option to select pods based on label. Kail takes tailing Kubernetes logs to a whole new level.

For Homebrew users, it’s available via brew install boz/repo/kail. When executed without any arguments it tails logs for all containers in the cluster which is probably not what you want unless your cluster is very quiet! There are, however, flags to let you filter not just on pod, container, and label, but also namespace, deployment, replica set, ingress, service, or node. Flags of the same type are ORed together, different flags are ANDed. And that’s pretty much all there is to it but anyone who finds themselves watching the logs of any moderately complex application will wonder how they lived without it!

Finally, I get to write that blog post on kustomize! kustomize is yet another tool attempting to solve the problem of how to make Kubernetes configuration re-usable. Unlike, say, Helm, kustomize allows configuration to be overridden at consumption time without necessarily having allowed for it when the configuration was originally produced. This is great if you are attempting to re-use someone else’s configuration. On the flip-side, you might prefer to use something like Helm if you actually want to limit the points of variability e.g. to ensure standardization across environments or applications.

You know the drill by now: the go binary CLI can be obtained via brew install kustomize. There is one main command and that is kustomize build. That expects to be pointed at a directory or URL containing a kustomization.yaml file. Running the command outputs the required Kubernetes resources to standard output where they can then be piped to kubectl if desired.

The kustomization.yaml can contain the following directives:

• namespace – to add a namespace to all the output resources
• namePrefix – to add a prefix to all the resource names
• commonLabels – to add a set of labels to all resources (and selectors)
• commonAnnotations – to add a set of annotations to all resources
• resources – an explicit list of YAML files to be customized
• configMapGenerator – to construct ConfigMaps on the fly
• secretGenerator – to construct Secrets via arbitrary commands
• patches – YAML files containing partial resource definitions to be overlayed on resources with matching names
• patchesJson6902 – applies a JSON patch that can add or remove values
• crds – lists YAML files defining CRDs (so that, if their names are updated, resources using them are also updated)
• vars – used to define variables that reference resource/files for replacement in places that kustomize doesn’t handle automatically
• imageTags – updates the tag for images matching a given name

That’s a pretty comprehensive toolbox for manipulating configuration. The only directive I didn’t mention was bases with which you can build a hierarchy of customizations. The prototypical example given is of a base configuration with different customizations for each deployment environment. Note that you can have multiple bases, so aws-east-staging might extend both aws-east and staging.

One of the refreshing things about kustomize is that it explicitly calls out a set of features that it doesn’t intend to implement. This introduces the only other command that the CLI supports: kustomize edit. Given that one of the stated restrictions is that kustomize does not provide any mechanism for parameterising individual builds, the intent of this command is to allow you to script modifications to your kustomization.yaml prior to calling build.

It’s worth noting that kustomize can be used in combination with Helm. For example, you could run helm template and then use kustomize to make additional modifications that are not supported by the original chart. You can also use them in the reverse order. The Helmfile docs describe how to use Helmfile’s hooks to drive a script that will use kustomize to construct the required YAML, but then wrap it in a shell chart so that you get the benefit of Helm’s releases.

I wanted to take a look at ksonnet and kustomize  but felt I should write about what I know best first, and that’s Helm. All three tools are trying to tackle the same basic problem: enabling the re-use of Kubernetes configuration where typically we want some level of customisation each time we use the configuration, whether that’s to reflect the latest deployment or the environment we’re deploying to.

The starting point for Helm is a client binary called, unsurprisingly, helm, and is available from brew as kubernetes-helm. The current version of Helm also has a server-side component called Tiller and that is deployed by executing helm init --wait (the wait flag indicates to wait until the Tiller pod has started before returning). Note that Helm is pretty picky about having matching versions of client and server.

In keeping with Kubernetes nautical theme, Helm’s unit of packaging is the chart. A new chart called test can be easily constructed with the command helm create test. This results in a directory called test with the following contents:

Chart.yaml contains some basic meta-data about the chart such as name, description and version. Note that Helm draws a distinction between the version of the chart and the version of the application that the chart deploys. We’ll come back to the empty charts directory later. The templates directory contains the YAML files that we’d expect to find for a basic Kubernetes application and these can be extended and/or replaced as needed. As the directory name suggests though, and this is the key to reuse, the files are actually templates. If we look in deployment.yaml we’ll see the mustache syntax used to support substitution:

We can see that it’s going to use the name from the Chart.yaml for the container name. The default contents for the Values fields are taken from values.yaml and typically this also contains comments describing the intent of each value. The templates use Go template syntax and support the sprig functions along with a few Helm-specific extensions. The syntax is pretty rich and supports pretty much any manipulation you’re likely to want to perform on configuration. The _helpers.tpl file defines some helper functions that are used throughout the chart. Finally, the NOTES.txt contains text that is output when the chart is installed, typically providing usage instructions for the installed application. This file also supports templating.

The default generated chart deploys a pod with nginx in it. If we look in values.yaml we can see that, by default, it’s going to use the stable image tag. We can override this at install time. For example:

If we want to override lots of values then we can specify then via a YAML file instead. Indeed, we can specify multiple files so there might, for example, be one for production and another for us-east-1. The name here is optional but, at least for scripting, it’s easier not to use one of the generated names. Note that, although the release is deployed to a namespace (and Helm isn’t capable of tracking resources that explicitly target some other namespace), the name is scoped to the tiller instance (i.e. the cluster if you only have one tiller install).

There are others commands for operating on releases: delete, upgrade, rollback, list, status and get, all of which do pretty much what you’d expect of them. As the upgrade, rollback and history commands suggest, helm is tracking revisions of a release. Tip: if you’re in some CD pipeline and you don’t know whether you’re upgrading or installing for the first time, use helm upgrade --install and it will do the right thing. Something else to watch out for is the --wait option on an upgrade. This waits until you have at least replicas - maxUnavailable pods available. Given that these are both typically one by default, don’t be surprised when it doesn’t appear to be waiting!

We’ve just installed a chart off the local filesystem but, as with any decent package manager, Helm has the concept of a repository. By default, the helm CLI is configured with the stable and incubator repositories. You can search these with helm search or head over to Kubeapps for a shiny catalog. These show the real power of Helm when, with a simple helm install stable/wordpress, you can re-use someone else’s hard work in defining the best practise for deploying WordPress on Kubernetes. You can add other repositories (e.g. that for Jenkins X) and you can create your own, either via a simple file server or, a read-write repository using ChartMuseum and the Monocular UI.

The packages themselves are simply .tgz files although the helm package command also supports things like signing the package. I said I’d come back to the charts directory and the WordPress chart is a good example of how this can be used. The WordPress chart actually also deploys the MariaDB chart to provide persistent storage. This is achieved by placing a requirements.yaml in the root of the chart that specifies the dependency. Dependencies can be fetched with helm dependency update or resolved at packaging time with the --dependency-update flag.

Jenkins X makes use of this sub-chart capability in the way it deploys environments. The repo for each environment contains a Helm chart which specifies as dependencies all of the application charts to deploy in to the environment. One downside of this approach is that you then only see a Helm release for the entire environment. Another issue with sub-charts relates to the way their values are overridden. The values for a sub-chart can be overridden by prefixing with the name or alias given in requirements.yaml but there is no good way to get the same value in to multiple sub-charts. If you have control over the sub-charts you can write them to retrieve values from a global scope but that doesn’t help if you’re trying to re-use someone else’s efforts.

Helm provides lots of other goodness. You can annotate resources with hooks so that they are run pre or post install, upgrade, rollback or delete. There is even a crd-install annotation to ensure that your CRDs are created before other resources attempt to use them. You should also know helm lint and helm test. The latter executes resources in the chart with the test-success or test-failure hook annotations. You can use these to provide a detailed check of whether a release was successfully deployed.

An overview of Helm wouldn’t be complete without a reference to Helm v3. At the beginning I indicated that the current version deploys a server-side component. That is particularly problematic when it comes to security as, even if you enable mutual-TLS, any user that can connect can perform any action supported by the service account under which Tiller is running, losing you all of Kubernetes’ RBAC benefits. You can mitigate the problem by installing a Tiller per namespace but you’re still left managing certificates. IBM Cloud Private implemented their own Tiller to do identity assertion in order to overcome this problem. Jenkins X now supports using helm template to render charts client-side before deploying with kubectl. This means that you don’t get any of the Helm release management but then that is handled by Jenkins X anyway. Helm 3 promises to do away with Tiller but still share release information server-side via CRDs. Sadly as James alludes to in his blog post, there’s not a lot of public progress being made on Helm 3 at the moment.

Next on the list of projects utilised by Jenkins X (and this is a theme that could run and run) is Google’s Skaffold. There is an intersection between the capabilities of Skaffold and Draft. Skaffold does not provide anything like packs (this is where Jenkins X uses Draft) so you need to provide your own Dockerfile and Kubernetes configuration (raw YAML, kustomize templates, and Helm charts are supported). Both projects, however, aim to simplify the process of building an image and deploying it to a Kubernetes cluster to speed iterative development of applications that have hard dependencies on a Kubernetes environment, or the services running therein.

In the Skaffold case, a skaffold run will do a one-time build and deploy and a skaffold dev will continuously monitor the filesystem to determine when to rebuild and update. As previously discussed, the value of being able to do this versus needing a smart incremental deployment from an IDE is very much dependent on how quick that rebuild process is for your application language/runtime of choice. As with Draft, it has the ability to allow you to skip pushing to a registry when you’re working with a local cluster.

So what does Skaffold offer that Draft doesn’t? Principally, that it is not designed to be used solely at development time. The idea is that the same skaffold run may also be used as part of your continuous deployment pipeline. If you’re a GCP user, this extends to capabilities like using Google Cloud Builder or Kaniko rather than a simple Docker build and, of course, interaction with a registry.

As this annotated skaffold.yaml shows, it has a few other neat tricks. You have lots of flexibility over the tagging scheme used for images: SHA, git commit ID, timestamp, or a Golang template. For a Docker build, you can specify build arguments and cache images. You can even configure a bazel build instead of a Docker build.

One thing to watch is that you adhere to the convention used for substituting the names of the built images. When using Helm, for example, the default behaviour is to pass the combination of image repository/name and tag as a single value. If you chart is using the default Helm convention of separate .repository and .tag values then you need to specify a different imageStrategy. If your chart expects a three-way split between repository, name, and tag, then you’re on your own!

Continuing on the theme of technologies used by Jenkins X under the covers, this post is going to take a look at ksync. In Jenkins X, this project is used to implement the ‘DevPods‘ capability. It’s a pretty thin veneer over the top of what is provided by ksync though. As the name suggests, the project performs synchronisation, in particular, bi-directional synchronisation between a directory on your laptop and a pod in a Kubernetes cluster. The aim is to allow you to code locally in your favourite editor but run that code in-cluster where you have access to a full Kubernetes environment, including any other services you might be running.

The workflow is simple: a <code>ksync init sets up your local environment and deploys a daemon set in the cluster to get access to the file system of containers. Next, a ksync watch runs a process locally to monitor your laptop. The last step is to issue a ksync create to form a mapping between a local directory and a directory in a container (or potentially multiple containers that match the selector). And that’s all there is to it: make a change locally and it’s reflected in the container; see files written by the container appear on your local disk.

ksync is, in turn, using syncthing to perform the actual file synchronisation. In addition, it provides the option to restart the application container following a sync (for example, if the application process is not dynamically reloading modified files).

In practise, it works well and certainly supports a much more iterative development experience than you would get if you relied upon a Docker build from something like <code>draft up</code>, even assuming you had optimised your Docker build so it just needed to copy in the application files. As when running an application locally, all of this works best for interpreted languages like PHP, Python or Node. For a language like Java, you need to get the compiled application (or classes) in to the synced directory and that is unlikely to give you the same experience that you can get using an IDE capable of hot-reloading. That’s something that Microclimate looked to address…