Book Review: Mastering Docker and Monitoring Docker

June 19th, 2016

Mastering DockerThe films on my flight to the US this week weren’t much to write home about so I ploughed through some of my Safari Queue backlog. Two of the Docker related books on my queue were from Packt Publishing.

The first was Mastering Docker by Scott Gallagher which I’ll confess that I only skim read. Parts of the book were already showing their age even though it was only published in December 2015 but that’s inevitable in such a fast-moving area. More problematic was that I found myself disagreeing with the author so much in just the first couple of chapters that I couldn’t believe anything I read after that. By way of just one example: the author asserts that commands are chained together in Dockerfiles to speed build times and seems to suggest that items added in one layer may be removed in another to keep the image size down. The structure of the book is also poor with material repeated throughout. It certainly doesn’t contain the depth to provide mastery in anything!

Monitoring DockerThe second book from the same stable was Monitoring Docker by Russ McKendrick. Maybe it’s just because this is an area that I know less about but I got on better with this book. It covers use of the Docker API (top, stats and logs), cAdvisor and aggregation with Prometheus. The author covers Zabbix in some detail which, I must confess, is a tool that I’d never even heard of, before a gushing endorsement for Sysdig. The book provides an overview of SaaS options (Sysdig Cloud, Datadog and New Relic) before closing with a detailed walk-through of setting up a containerized ELK stack for log aggregation. There are certainly options that the book didn’t cover but I felt it provided good coverage of the considerations to allow the reader to make informed decisions about what they should look for in a monitoring solution.

I also took a look at the early release of Kelsey Hightower’s Kubernetes: Up and Running from O’Reilly. It shows promise but, as it’s not due for release until October, there wasn’t enough content there yet to give a review.

WebSphere Liberty and IBM Containers: Part 3

June 18th, 2016

Scaling up

In the first two posts in this series we covered the basics of starting a WebSphere Liberty on IBM Containers via the browser and then using the command line to deploy an application.

We’ve already seen some of the value-add that comes out of the box when running under IBM Containers. For example, at no point have we needed to be concerned with the underlying infrastructure on which the containers are running (beyond selecting a region). When we created an image it was scanned automatically for vulnerabilities. Each container was allocated its own private IP address accessible from any other container running in the same space – no need to set up and configure overlay networking here. We had control over whether we also wanted to assign a public IP and, if so, what ports should be exposed there. We also had easy access to metrics and standard out/error from the container.

So far we’ve only deployed a single container though. What happens when we hit the big time and need to scale up our application to meet demand? When we created our first container via the UI, you may remember that the Single option was selected at the top. Let’s go back and try out the Scalable alternative. From the catalog, navigate through Compute and Containers (remember that these instructions are based on the new Bluemix console). Select our existing demo image. Next, select the Scalable button at the top and give the container group a name. By default you’ll see that our group will contain two instances.

Rather than having a single IP associated with a container, this time we are asked to give a host name (on the domain by default) for the group. Requests arriving at this host name will be load-balanced across the container instances in the group (reusing the gorouter from Cloud Foundry). One nice bonus of this approach is that it doesn’t eat in to our quota of public IPs! As the host name needs to be unique within the domain, I tend to include my initials as you’ll see in the screenshot below. Select 9080 as the HTTP port and then click Create.

Container group creation

Once the containers have started, the dashboard should show that you have two instances running:

Running instances

Right-click on the route shown in the Group details section and open it in a new tab. This should take you to a Liberty welcome page and, if you add the myLibertyApp context root, you should be able to see the application again. If you hit refresh a few times, although you won’t be able to tell with this application, your requests will be load-balanced across the two instances. If you return to the dashboard and switch to the Monitoring and Logs tab you can switch between the output for the instances and should, for example, be able to see the spike in network usage on the two containers when you made the requests.

If you return to the Overview tab you will see that there are plus and minus symbols either side of the current number of instances. These can be used to manually scale the number of instances. Click the + icon, click Save, and watch the creation of a new container in the group.

Manual scaling is all very well but it would be better if the number of instances scaled automatically up and down as required. If you’re deploying your containers in the London region then you’ll notice an extra tab at the top of the dashboard labelled Auto-Scaling. It’s only available in the London region at the moment because the service is still in beta (and so things may change a bit from what I’m describing here). Having selected this tab, click the plus icon labelled Create policy. Give the policy a name default and set the minimum and maximum instance values to 1 and 3. Add two CPU usage rules to scale up and down the number of instances as shown in the following diagram and then hit Create. Finally, select Attach to activate the policy for this scaling group.


If you click the Auto-Scaling History tab you should see that a scaling action has taken place. We originally scaled up manually to 3 instances but, as the CPU usage is below our 60% limit, the number gets scaled down by one. If you wait another 5 minutes (the cool down period we specified), then you’ll see it get scaled down again to our minimum of 1.

Scaling history

And that concludes our tour of the scaling options in IBM Containers!

WebSphere Liberty on IBM Containers: Part 2

May 30th, 2016

Deploying an Application

In the first part of this series we looked at how to get started running a WebSphere Liberty image in IBM Containers using the Bluemix console.  The container was just running an empty Liberty server. In this post we’ll look at building and deploying an image that adds an application. I was originally intending to stick to using the browser for this post but I think it’s actually easier to do this from the command line. I’m going to assume that you already have Docker installed locally, either natively on Linux, via Docker Machine, or via the Docker for Mac/Windows beta.

First off we need an application to deploy and, just for novelty, I’m going to use the Liberty app accelerator to generate one. Select Servlet as the technology type and then, contrary as it may seem, select Deploy to Local and not Deploy to Bluemix. The latter option currently only supports deploying to the Instant Runtimes (Cloud Foundry) side of Bluemix. Finally, give your project a name and click Download Now.

Liberty App Accelerator

Unpack the zip file you downloaded and change to the top directory of the project. The app is built using maven. Perhaps you already have maven installed but this is a Docker blog post so we’re going to use the maven image from Docker Hub to build the app as follows:

$ docker run –rm -v $(pwd):/usr/src/mymaven \
    -w /usr/src/mymaven/myProject-application maven mvn clean package

This mounts the project on to a container running the maven image and runs the command mvn clean package in the myProject-application directory. Note: if you were doing this repeatedly you’d probably want to mount a maven cache in to the image as well and not download everything each time)

In the myProject-application/target directory you should now find that you have a file myArtifactId-application-1.0-SNAPSHOT.war. Copy this in to a new empty directory so that when we execute a Docker build we don’t end up uploading lots of other cruft to the Docker engine. Using your favourite editor, add the following Dockerfile to the same directory:

FROM websphere-liberty:webProfile7
COPY myArtifactId-application-1.0-SNAPSHOT.war /config/dropins

We have two choices now, we can either build a Docker image locally and then push that up to the IBM Containers registry, or we can build the image in IBM Containers. We’ll go for the latter option here as it involves pushing less bytes over the network.

There’s one niggle today that, to access your IBM Containers registry, you need to log in first using the Cloud Foundry CLI and IBM Containers plugin. We’re going to play the containerisation trick again here. Run the following commands to build an image with the CLI and plugin:

$ docker build -t cf

Ideally I’d run this image as a stateless container but getting the right state written out to host in to the .cf, .ice and .docker directories is a bit finicky. Instead, we’re going to mount our current directory on to an instance of the image and perform the build inside:

$ docker run -it –rm $(pwd):/root/build cf
$ cd /root/build
$ cf login -a

$ cf ic login
$ cf ic build -t demo .

Now we’re ready to run an instance of your newly built image. At this point you could switch back the to the UI but lets keep going with the command line. We’ll need to refer to the built image using the full repository name, including your namespace:

$ ns=$(cf ic namespace get)
$ cf ic run –name demo -P$ns/demo

By default, containers are only assigned a private IP address. In order to access our new container we’ll need to request and assign a public IP. The cf ic ip command unfortunately returns a human friendly message, not a computer friendly one, hence the need for the grep/sed to retrieve the actual IP:

$ ip=$(cf ic ip request | grep -o ‘”.*”‘ | sed ‘s/”//g’)
$ cf ic ip bind $ip demo

Lastly, we can list the port and IP to point our browser at:

$ cf ic port demo 9080

Adding the root context myLibertyApp should give use the welcome page for the starter app.

Starter Welcome Page

Congratulations, you’ve successfully deployed an application to IBM Containers! In the next post in this series we’ll look at some of the additional features that the service provides, such as scaling groups and logging.

Climbing and Cones

May 29th, 2016

Emma ClimbingEmma’s had a prolonged birthday this past week. Last weekend was her party with Emma and seven friends from school heading to the climbing wall at St Mary’s Leisure Centre in Southampton. (The traffic getting in to Southampton was a nightmare. We’re not sure whether it was the presence of the world’s largest cruise liner or Peter Andre that was pulling in the crowds!) After the initial briefing Emma scampered up to the top of the wall without too much difficultly. If we’d placed bets on which of the other girls would be slightly more reluctant then we’d have made a healthy profit! The instructor was very good though, encouraging them on but not pushing them if it was clear they just wanted to come back down. Everyone enjoyed the last part spent in the bouldering room.

Emma's Birthday CakeThe shape of the cake was a clue to the second destination for the party: Sprinkles Gelato. Unfortunately we weren’t treated to the same glorious weather we’d had the previous weekend when we’d walked the route but there wasn’t too much complaining about the rain. We had a table booked and eventually managed to get everyone to decide what they wanted to eat. Having seen the size of a two-scoop cone the previous week, we limited them all to a single scoop and toppings  which made it quite a cheap visit! They shop was also very good in helping us navigate the various allergies of one of Emma’s friends so the EpiPen wasn’t needed.

Emma and CakeEmma’s birthday itself came round later in the week. Her school lets pupils wear non-uniform on their birthday which meant Emma could try out one of her new dresses. It’s a nice way to make them feel that extra bit special. One of Emma’s other presents from us will take slightly longer to get any use from us. The crystal growing set suggests using a well ventilated basement room for the experiments! The biggest smile on Emma’s face though was when she came back from gymnastics that evening having passed her badge 2.

WebSphere Liberty on IBM Containers: Part 1

May 16th, 2016

Getting Started

I wanted write a few blog posts as I explored deploying our WebSphere Liberty Docker image on to third-party cloud platforms but this didn’t really feel fair given that I haven’t written anything here about the options using IBM’s cloud platform! So, to kick things off, here’s a series of posts on using Liberty with the IBM Containers service in Bluemix. In this post I’m going to focus on using the web UI to get a simple Liberty container up and running. In subsequent posts I’ll cover topics such as deploying an application, scaling applications and using the command line.

I’m also going to use the new console for the instructions/screenshots in this post. If you don’t already have an account, use the sign-up button at the top right of the console to get a free 30-day trial.

You can use WebSphere in a number of different ways in Bluemix: web applications deployed as Cloud Foundry applications will use the Liberty runtime and it’s also possible to stand-up WAS ND topologies for both Liberty and the traditional application server in VMs. Here we’re going to focus on containers.

Once logged in, select the Compute icon and then the Containers tab. Then click the + icon top-right to create a new container. You should find that the private registry associated with your account is already populated with a number of IBM-provided images as shown in the following screen shot. Select the ibmliberty image. This image adds a few layers on top of the websphere-liberty image from Docker Hub to make it run more cleanly in IBM Containers (see the Dockerfile in the docs for full details).

Select container image

You will now see a drop-down on the left which allows you to select the image tag to use. By default the latest tag will give you the full Java EE7 profile image. Select the webProfile7 tag to get a lighter-weight image. Beneath the tag you’ll an area entitled Vulnerability Assessment which, if we’ve been doing our properly, should say Safe to Deploy. Assessments are updated regularly as new CVEs are announced and we therefore regularly refresh the image to make sure that it stays clean.

Vulnerabilty Advisor

At the top of the page, you’ll see that the Single option is selected to create just a single container instance, we’ll return later to see how the Scalable option works. Give the container a name. Leave the container size at Micro but select the option to Request and Bind Public IP.  All containers get an IP address on a private network and are accessible by other containers in the same space but to be accessible externally they must be assigned a public IP.

Create a container

Now click the Create button at the bottom of the page. After a minute or two the status of the container should change to Running.

Container running

Switch to the Monitoring and Logs tab. The container level metrics around CPU/memory/network usually take some time to appear but if you select the Logging tab you should be able to see the messages output by Liberty as the container started. Tip: you can use the ADVANCED VIEW button on these tabs to open up Grafana and Kibana views on to these metrics and logs respectively if you want to do more detailed analysis.

Container logs

Finally, switch back to the Overview tab. If you scroll down to the Container details section, you’ll find a list of ports. If you click on 9080, this will open a new browser tab using that port and the public IP address assigned to the container. With the default image, that should show the Liberty welcome page.

Liberty welcome page

Congratulations – you have run WebSphere Liberty on IBM Containers! The astute among you will, however, have noticed that it’s not running any application. That’ll be the subject of the next blog post…

Two Days with Uncle Bob

May 13th, 2016

I’ve been lucky enough to spend the past two days in the presence of Robert C. Martin a.k.a. Uncle Bob for his Clean Code workshop at Skills Matter in London. I arrived at the very last-minute having decided to try out a Santander Cycle for the first time, only to discover that the promised docking station next to Code Node has disappeared under road works.

The workshop got off to a slow start as we spent over an hour going round the room indicating how long we’d been programming and what languages we’d used. This seemed to be just so that we could observe that, as well as a fair smattering of those in the first 5-10 years, there were many of us beyond our first flush of youth! That, and so that Bob could provide a potted history of various programming languages. This was a general theme for the course with Bob prefixing each module with an interlude on some aspect or other of astro or nuclear physics!

As a consequence of the above, if attendees were expecting to learn all of the content of the Clean Code book during the workshop then they’d leave disappointed. (Although a copy of the book was included in the exorbitant course fee.) What you did get was a good understanding of the values that underlie the practices covered by the book, the opportunity to watch the master at work refactoring and plenty of opportunity to ask questions first-hand.

I shan’t attempt to repeat the material but there were two main a-ha moments for me. The first was that, as a result of not doing test-first development, we’ve ended up with unit tests focused on each individual method. This has made them extremely fragile to refactoring which, in turn, means that either the tests quickly fall by the wayside or that refactoring just doesn’t happen. And that leads to the second learning point: to refactor mercilessly, extracting functions until every function does just one thing. I had fallen in to the trap of seeing the myriad of small methods as adding to the complexity. Bob’s argument is that, well-named, the methods allow you to quickly gained an understanding of what the code is doing at every level of abstraction. Based on the one exercise that we did during the workshop, I’d say that pairing is immensely helpful in determining what makes a good name. Anyway, lots to share back with my colleagues!

As an aside: I can heartily recommend the Hawksmoor Seven Dials if you are a fan of steak (although it has to be said that the corporate expense limit would have just about covered the meat on my plate!).

British Birthday

May 4th, 2016

Christine @ Brown Clee British ChampsFor the Bank Holiday weekend we headed to the British Orienteering Champs at Brown Clee in Shropshire. We arrived in glorious sunshine on Saturday morning but this soon turned to hail as we climbed up the hill to the assembly area. We were very grateful that a club mate had already erected the club tent as the hail came and went for most of the afternoon.

I had an early start and it has to be said that I had a pretty poor run in my first year at M40. Although 7th place in the results doesn’t sound too bad, I was over half an hour behind Georgie Best with ten minutes of that loss being attributable to just two controls. (With 27 controls in 9.2k there was plenty of opportunity for losing time!) Christine failed to repeat her W16 win at Brown Clee, faring marginally better than me but still ending up ten minutes down on her course winner. Emma had the best result of the day, coming third on the white course in a time that would have netted her a win on W10B if she had the confidence to go out on her own.

With rain forecast for most of Sunday, Christine and I returned for the relays without the children. I’m glad to say that we both had better runs. Christine returned first on the Women’s Short just as I set out on first leg on the Men’s Short. I ran alone for much of the course being caught just at the end and finishing just a few seconds down in second place. The commentary team had been primed to wish me a Happy Birthday as I came down the run-in! My team finished in fifth place whilst Christine, sadly, did not have a complete team.

On the Monday Emma ran again, completing the hilly 2km at Devauden XC races in 11:16. Christine and I were happy to sit the 10k out this year and watch her Mum run instead!

Active Weekend

April 24th, 2016

Emma RunningDuncan RunningIt’s been an active weekend in the Currie household. On Saturday morning we made the trek over to Staunton Country Park for the second in SOC’s Solent Summer Series. Buoyed by just receiving the SOC Junior Women’s trophy for 2015 (although she was quick to point out that she didn’t have much competition, Emma was happy to go out on the yellow with Christine and a grumpy Duncan in tow. I went out on the light green which was more than a little muddy in places. Rob Finch managed to turn the tables from Fleming Park pipping me to first place by 14 seconds.

Christine after half marathonFrom there we headed in to Southampton for the ABP Southampton Fun Run. Christine set off on the 1 mile course with both children but Duncan got a stitch and Emma ran off and left them towards the end. The field was smaller than I had expected given the potential catchment area for the race. Christine then ran again in the half marathon on Sunday finishing in a chip time of 1:40:54. Not bad given the lack of training she’s done due to injury.