David Currie

The sample for Microservice Builder is intended to run on top of the Microservice Builder fabric and also to utilize the ELK sample. As such, the Kubernetes configuration for the microservice pods all bind to a set of resources (secrets and config-maps) created by the Helm charts for the fabric and ELK sample. The slightly annoying thing is that the sample would work perfectly well without these (you just wouldn’t get any logging to the ELK stack) only, as we shall see in a moment, deployment fails if the fabric and ELK sample have not already been deployed. In this post we’ll explore a few possibilities as to how these resources could be made optional.

I’m going to assume a minikube environment here and we’re going to try to deploy just one of the microservices as follows:

git clone git@github.com:WASdev/sample.microservicebuilder.session.git
cd sample.microservicebuilder.session
eval $(minikube docker-env)
docker build -t microservice-session .
kubectl apply -f manifests

If you then perform a kubectl describe  for the pod that is created you’ll see that it fails to start as it can’t bind the volume mounts:

FirstSeen	LastSeen	Count	From			SubObjectPath	Type		Reason			Message
---------	--------	-----	----			-------------	--------	------			-------
1m		1m		1	default-scheduler			Normal		Scheduled		Successfully assigned microservice-session-sample-3309272859-6241s to minikube
1m		1m		1	kubelet, minikube			Normal		SuccessfulMountVolume	MountVolume.SetUp succeeded for volume "default-token-nfzj9"
1m		18s		8	kubelet, minikube			Warning		FailedMount		MountVolume.SetUp failed for volume "liberty-config" : configmaps "liberty-logging-config" not found
1m		18s		8	kubelet, minikube			Warning		FailedMount		MountVolume.SetUp failed for volume "truststore" : secrets "mb-truststore" not found
1m		18s		8	kubelet, minikube			Warning		FailedMount		MountVolume.SetUp failed for volume "keystore" : secrets "mb-keystore" not found

Elsewhere in the output though you’ll see a clue to our first plan of attack:

Volumes:
  keystore:
    Type:	Secret (a volume populated by a Secret)
    SecretName:	mb-keystore
    Optional:	false

Doesn’t that optional  flag look promising?! As of Kubernetes 1.7 (and thanks to my one-time colleague Michael Fraenkel) we can mark our usage of secrets and config-maps as optional. Our revised pod spec would now look as follows:

    spec:
      containers:
      - name: microservice-session
        image: microservice-session:latest
        ports:
          - containerPort: 9080
        imagePullPolicy: IfNotPresent
        env:
          - name: MB_KEYSTORE_PASSWORD
            valueFrom:
              secretKeyRef:
                name: mb-keystore-password
                key: password
                optional: true
          - name: MB_TRUSTSTORE_PASSWORD
            valueFrom:
              secretKeyRef:
                name: mb-truststore-password
                key: password
                optional: true
        volumeMounts:
        - name: keystore
          mountPath: /etc/wlp/config/keystore
        - name: truststore
          mountPath: /etc/wlp/config/truststore
        - name: liberty-config
          mountPath: /config/configDropins
        readinessProbe:
          httpGet:
            path: /sessions/nessProbe
            port: 9080
      volumes:
      - name: keystore
        secret:
          secretName: mb-keystore
          optional: true
      - name: truststore
        secret:
          secretName: mb-truststore
          optional: true
      - name: liberty-config
        configMap:
          name: liberty-logging-config
          items:
            - key: keystore.xml
              path: defaults/keystore.xml
            - key: logging.xml
              path: defaults/logging.xml
          optional: true

And lo and behold, with that liberal sprinkling of optional attributes we can now successfully deploy the service without either the fabric or ELK sample. Success! But why stop there? All of this is boilerplate that is repeated across all our microservices. Wouldn’t it be better if it simply wasn’t there in the pod spec and we just added it when it was needed? Another new resource type in Kubernetes 1.7 comes to our rescue: the PodPreset. A pod preset allows us to inject just this kind of configuration at deployment time to pods that match a given selector.

We can now slim our deployment down to the bare minimum that we want to have in our basic environment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: microservice-session-sample
spec:
  replicas: 1
  template:
    metadata:
      labels:
        name: session-deployment
        runtime: liberty
    spec:
      containers:
      - name: microservice-session
        image: microservice-session:latest
        ports:
          - containerPort: 9080
        imagePullPolicy: IfNotPresent
        readinessProbe:
          httpGet:
            path: /sessions/nessProbe
            port: 9080

Note that we have also added that runtime: liberty  label to the pod which is what we’re going to use to match on. In our advanced environment, we don’t want to be adding the resources to every pod in the environment, in particular,  we don’t want to add it to those that aren’t even running Liberty. This slimmed down deployment works just fine, in the same way that the optional version did.

Now, what do we have to do to get all of that configuration back in an environment where we do have the fabric and ELK sample deployed? Well, we define it in a pod preset as follows:

apiVersion: settings.k8s.io/v1alpha1
kind: PodPreset
metadata:
  name: logging-pod-preset
spec:
  selector:
    matchLabels:
      runtime: liberty
  env:
    - name: MB_KEYSTORE_PASSWORD
      valueFrom:
        secretKeyRef:
          name: mb-keystore-password
          key: password
    - name: MB_TRUSTSTORE_PASSWORD
      valueFrom:
        secretKeyRef:
          name: mb-truststore-password
          key: password
  volumeMounts:
  - name: keystore
    mountPath: /etc/wlp/config/keystore
  - name: truststore
    mountPath: /etc/wlp/config/truststore
  - name: liberty-config
    mountPath: /config/configDropins
  volumes:
  - name: keystore
    secret:
      secretName: mb-keystore
  - name: truststore
    secret:
      secretName: mb-truststore
  - name: liberty-config
    configMap:
      name: liberty-logging-config
      items:
        - key: keystore.xml
          path: defaults/keystore.xml
        - key: logging.xml
          path: defaults/logging.xml

Note that the selector is matching on the label that we defined in the pod spec earlier. Now, pod presets are currently applied by something in Kubernetes called admission control and, because they are still alpha, minikube doesn’t enable the admission controller for PodPresets by default. We can enable it as follows:

minikube start --extra-config=apiserver.Admission.PluginNames=\
  NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,\
  DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds,PodPreset

(Note that, prior to minikube v0.21.0 this property was called apiserver.GenericServerRunOptions.AdmissionControl, a change that cost me half an hour of my life I’ll never get back!)

With the fabric, ELK sample and pod preset deployed, we now find that our pod regains its volume mounts when deployed courtesy of the admission controller:

    Environment:
      MB_KEYSTORE_PASSWORD:	<set to the key 'password' in secret 'mb-keystore-password'>	Optional: false
      MB_TRUSTSTORE_PASSWORD:	<set to the key 'password' in secret 'mb-truststore-password'>	Optional: false
    Mounts:
      /config/configDropins from liberty-config (rw)
      /etc/wlp/config/keystore from keystore (rw)
      /etc/wlp/config/truststore from truststore (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-tl2hz (ro)

Pod presets are tailor-made for this sort of scenario where we want to inject secrets and config maps but even they don’t go far enough for something like Istio where we want to inject a whole new container into the pod (the Envoy proxy) at deployment time. Admission controllers in general also have their limitations in that they have to be compiled into the API server and, as we’ve seen, they have to be specified when the API server starts up. If you need something a whole lot more dynamic that take a look at the newly introduced initializers.

One last option for those who aren’t yet on Kubernetes 1.7. We’re in the process of moving our generated microservices to use Helm and in a Helm chart template you can make configuration optional. For example, we might define a logging  option in our values.yaml with a default value of disabled , and then we can define constructs along the following lines in our pod spec:

    spec:
      containers:
      - name: microservice-session
        image: microservice-session:latest
        ports:
          - containerPort: 9080
        imagePullPolicy: IfNotPresent
        env:
          {{- if eq .Values.logging "enabled"}}
          - name: MB_KEYSTORE_PASSWORD
            valueFrom:
              secretKeyRef:
                name: mb-keystore-password
                key: password
          {{- end}}

Then all we’ve got to do when we’re deploying to our environment with the fabric and ELK sample in place is to specify an extra –set logging=enabled  on our helm install. Unlike the pod preset, this does mean that the logic is repeated in the Helm chart for every microservice but it certainly wins on the portability stakes.