The sample for Microservice Builder is intended to run on top of the Microservice Builder fabric and also to utilize the ELK sample. As such, the Kubernetes configuration for the microservice pods all bind to a set of resources (secrets and config-maps) created by the Helm charts for the fabric and ELK sample. The slightly annoying thing is that the sample would work perfectly well without these (you just wouldn’t get any logging to the ELK stack) only, as we shall see in a moment, deployment fails if the fabric and ELK sample have not already been deployed. In this post we’ll explore a few possibilities as to how these resources could be made optional.
I’m going to assume a minikube environment here and we’re going to try to deploy just one of the microservices as follows:
|
|
git clone git@github.com:WASdev/sample.microservicebuilder.session.git cd sample.microservicebuilder.session eval $(minikube docker-env) docker build -t microservice-session . kubectl apply -f manifests |
If you then perform a
kubectl describe for the pod that is created you’ll see that it fails to start as it can’t bind the volume mounts:
|
|
FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 1m 1m 1 default-scheduler Normal Scheduled Successfully assigned microservice-session-sample-3309272859-6241s to minikube 1m 1m 1 kubelet, minikube Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "default-token-nfzj9" 1m 18s 8 kubelet, minikube Warning FailedMount MountVolume.SetUp failed for volume "liberty-config" : configmaps "liberty-logging-config" not found 1m 18s 8 kubelet, minikube Warning FailedMount MountVolume.SetUp failed for volume "truststore" : secrets "mb-truststore" not found 1m 18s 8 kubelet, minikube Warning FailedMount MountVolume.SetUp failed for volume "keystore" : secrets "mb-keystore" not found |
Elsewhere in the output though you’ll see a clue to our first plan of attack:
|
|
Volumes: keystore: Type: Secret (a volume populated by a Secret) SecretName: mb-keystore Optional: false |
Doesn’t that
optional flag look promising?! As of Kubernetes 1.7 (and thanks to my one-time colleague Michael Fraenkel) we can mark our usage of secrets and config-maps as optional. Our revised pod spec would now look as follows:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 |
spec: containers: - name: microservice-session image: microservice-session:latest ports: - containerPort: 9080 imagePullPolicy: IfNotPresent env: - name: MB_KEYSTORE_PASSWORD valueFrom: secretKeyRef: name: mb-keystore-password key: password optional: true - name: MB_TRUSTSTORE_PASSWORD valueFrom: secretKeyRef: name: mb-truststore-password key: password optional: true volumeMounts: - name: keystore mountPath: /etc/wlp/config/keystore - name: truststore mountPath: /etc/wlp/config/truststore - name: liberty-config mountPath: /config/configDropins readinessProbe: httpGet: path: /sessions/nessProbe port: 9080 volumes: - name: keystore secret: secretName: mb-keystore optional: true - name: truststore secret: secretName: mb-truststore optional: true - name: liberty-config configMap: name: liberty-logging-config items: - key: keystore.xml path: defaults/keystore.xml - key: logging.xml path: defaults/logging.xml optional: true |
And lo and behold, with that liberal sprinkling of optional attributes we can now successfully deploy the service without either the fabric or ELK sample. Success! But why stop there? All of this is boilerplate that is repeated across all our microservices. Wouldn’t it be better if it simply wasn’t there in the pod spec and we just added it when it was needed? Another new resource type in Kubernetes 1.7 comes to our rescue: the PodPreset. A pod preset allows us to inject just this kind of configuration at deployment time to pods that match a given selector.
We can now slim our deployment down to the bare minimum that we want to have in our basic environment:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: microservice-session-sample spec: replicas: 1 template: metadata: labels: name: session-deployment runtime: liberty spec: containers: - name: microservice-session image: microservice-session:latest ports: - containerPort: 9080 imagePullPolicy: IfNotPresent readinessProbe: httpGet: path: /sessions/nessProbe port: 9080 |
Note that we have also added that
runtime: liberty label to the pod which is what we’re going to use to match on. In our advanced environment, we don’t want to be adding the resources to every pod in the environment, in particular, we don’t want to add it to those that aren’t even running Liberty. This slimmed down deployment works just fine, in the same way that the optional version did.
Now, what do we have to do to get all of that configuration back in an environment where we do have the fabric and ELK sample deployed? Well, we define it in a pod preset as follows:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 |
apiVersion: settings.k8s.io/v1alpha1 kind: PodPreset metadata: name: logging-pod-preset spec: selector: matchLabels: runtime: liberty env: - name: MB_KEYSTORE_PASSWORD valueFrom: secretKeyRef: name: mb-keystore-password key: password - name: MB_TRUSTSTORE_PASSWORD valueFrom: secretKeyRef: name: mb-truststore-password key: password volumeMounts: - name: keystore mountPath: /etc/wlp/config/keystore - name: truststore mountPath: /etc/wlp/config/truststore - name: liberty-config mountPath: /config/configDropins volumes: - name: keystore secret: secretName: mb-keystore - name: truststore secret: secretName: mb-truststore - name: liberty-config configMap: name: liberty-logging-config items: - key: keystore.xml path: defaults/keystore.xml - key: logging.xml path: defaults/logging.xml |
Note that the selector is matching on the label that we defined in the pod spec earlier. Now, pod presets are currently applied by something in Kubernetes called admission control and, because they are still alpha, minikube doesn’t enable the admission controller for PodPresets by default. We can enable it as follows:
|
|
minikube start --extra-config=apiserver.Admission.PluginNames=\ NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,\ DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds,PodPreset |
(Note that, prior to minikube v0.21.0 this property was called apiserver.GenericServerRunOptions.AdmissionControl, a change that cost me half an hour of my life I’ll never get back!)
With the fabric, ELK sample and pod preset deployed, we now find that our pod regains its volume mounts when deployed courtesy of the admission controller:
|
|
Environment: MB_KEYSTORE_PASSWORD: <set to the key 'password' in secret 'mb-keystore-password'> Optional: false MB_TRUSTSTORE_PASSWORD: <set to the key 'password' in secret 'mb-truststore-password'> Optional: false Mounts: /config/configDropins from liberty-config (rw) /etc/wlp/config/keystore from keystore (rw) /etc/wlp/config/truststore from truststore (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-tl2hz (ro) |
Pod presets are tailor-made for this sort of scenario where we want to inject secrets and config maps but even they don’t go far enough for something like Istio where we want to inject a whole new container into the pod (the Envoy proxy) at deployment time. Admission controllers in general also have their limitations in that they have to be compiled into the API server and, as we’ve seen, they have to be specified when the API server starts up. If you need something a whole lot more dynamic that take a look at the newly introduced initializers.
One last option for those who aren’t yet on Kubernetes 1.7. We’re in the process of moving our generated microservices to use Helm and in a Helm chart template you can make configuration optional. For example, we might define a
logging option in our values.yaml with a default value of
disabled , and then we can define constructs along the following lines in our pod spec:
|
|
spec: containers: - name: microservice-session image: microservice-session:latest ports: - containerPort: 9080 imagePullPolicy: IfNotPresent env: {{- if eq .Values.logging "enabled"}} - name: MB_KEYSTORE_PASSWORD valueFrom: secretKeyRef: name: mb-keystore-password key: password {{- end}} |
Then all we’ve got to do when we’re deploying to our environment with the fabric and ELK sample in place is to specify an extra
--set logging=enabled on our helm install. Unlike the pod preset, this does mean that the logic is repeated in the Helm chart for every microservice but it certainly wins on the portability stakes.
The next leg of our Hungarian adventure took us to Orf?, a small tourist village, half an hour north of the city of Pécs (where we stopped briefly to stock up at Tescos!). Orf? sits beside a string of lakes on the edge of the Mecsek hills (the highest mountain in Hungary is only just over 1000m). We checked in to our shady apartment and then walked down to the village hall where registration for the Hungaria Cup was taking place. There was a bit of queue for the ‘foreign clubs’ and we didn’t help! We had underpaid and, to make the sums harder, I’d paid in Hungarian Forints but the transfer had arrived in Euros, and I wanted to pay the balance in Forints!
The next day, we walked to the assembly area for the orienteering and then promptly had to retrace half our steps to the start which was high on the hill above the apartment. Thankfully they had decided to let the open courses start whenever they liked so Emma and Duncan went to the start with Christine and set off together. They were on the taped course, a great idea which allowed them to either follow the tape the whole way (as they did on Day 1) or make the course significantly shorter by following the obvious shortcuts on the map (as they did on subsequent days). I struggled on the steep climbs in the heat but, at least travelling slowly, I didn’t waste much time on navigational areas. Afterwards, we cooled off in the aqua park by the lake which was free to competitors.
Day 2 had the same assembly area but we drove this time as, with starts after twelve, we didn’t want to walk there in the midday sun. Thankfully my course was 2km shorter but I still didn’t manage to break 10 min/k. Emma failed to punch one of the controls despite having been with Duncan but the organisers were sympathetic and reinstated her. After the day had cooled a little, we climbed up the lookout for views over the surrounding hills.
The assembly moved again for the last two days to the neighbouring village of Abaliget. Christine was off early and took the children with her. She improved her position again, finishing second. The children made it back before her though and even had an interview with the commentator. I had a late start and, after some early blunders, was caught four minutes by the leader of my course. I was pleased to be able to hang on to him for the middle section. We made a return trip to the aqua park afterwards.
The final day of the orienteering was a chasing start, or at least it was for Christine. My cumulative time was more than 40 mins behind the leader which meant just starting off at minute intervals. We were back with the sink holes again and I had a pretty clean run, finishing second on the day which brought me up to seventh overall. Christine was also second which meant she retained her third place overall and secured a place on the podium. Thankfully she didn’t win the 12 (screw fit) light bulbs the men got but we did have a bottle of wine and 3 litres of apple juice to drink before leaving the country! We took the cave tour afterwards which was an interesting experience given it was all in Hungarian. If nothing else, it was nice and cool.
RSS - Posts