Author Archive

Containerizing background processes

Thursday, September 22nd, 2016

The lifetime of a Docker container is tied to the lifetime of the PID 1 process executed when the container was started. WebSphere Liberty has a convenient server run command to run the application server in the foreground. Sadly, that’s not the case with the traditional WebSphere’s startServer.sh script which simply starts the server process in the background and then exits. To ensure that the container didn’t exit as well, we started out with a script something along the following lines:

where server1.pid is a file created by the server process (but not immediately, hence the initial sleep). That successfully kept the container alive but failed to allow it to shutdown cleanly! A docker stop, for example, would wait for the default timeout period and then kill the process. Not great for any in-flight transactions! The solution was simple enough, add a trap to catch any interrupt and issue the command to stop the server:

All was well with the world until we then enabled server security by default. Unfortunately with security enabled the stopServer.sh script requires credentials to be provided and there is no way to get those credentials to the script. The solution was to switch to sending the interrupt signal to the server process. I also disliked that initial sleep so I decided to retrieve the process ID via ps (something that’s safer in a container given the limited process tree) and then wait whilst the processes directory exists in /proc. The resulting code looked along the following lines:

Note the use of a function so that $PID is not evaluated at the point the trap is set up.
Another disadvantage with having the server process in the background is the lack of output in the container logs. I decided to rectify that whilst I was at it by adding calls to tail the server log files:

The significance of the tail parameters is as follows. The capital F indicates that the attempts to follow the log file should be retried. This ensures that we continue to follow the latest file when the logs roll over. The pid parameter ensures that the background tail processes exit along with the server process. The -n +0 indicates that the output should start at the beginning of the file so that entries output whilst the startServer.sh script is running are not lost. As previously noted, Docker preserves stderr across the remote API so we make sure to direct the output from SystemErr.log there.

Start of the season

Sunday, September 18th, 2016

Emma and Duncan at the finishThe orienteering season got under way for us yesterday with a SOC event on an area called Salisbury Trench (east of Janesmoor Pond). Christine was saving herself for the Hursley 10K on Sunday so just took the children round the yellow course. (Actually, the children plus the pictured bear which Duncan has on loan from school as ‘Star of the Week’. It’s slightly concerning that he’s been awarded this in only the second week of the school year as normally it seems to go to those in need of some encouragement at school! Anyway, I digress…)

I went round the blue course which was quite fun, particularly given that the paths had been (deliberately) left off the map. The bracken needs a little longer to die off properly but the area was still pretty runnable although, as Pete Bray demonstrated later in the day, it was actually runnable at a lot faster pace than I was doing! The controls had been hung fairly high for which I was grateful as my compass-work was a little rusty. My ankle was also playing up a bit – a reminder not to get too carried away. The one event a month that SOC puts on is about the right frequency without having to stray further afield.

Container Camp III

Saturday, September 10th, 2016

Container CampOn Friday I attended my third Container Camp UK. The venue had changed once more, this time taking up residence in the Picturehouse Central cinema by Piccadilly Circus. As with last year, this meant comfy seats which, contrary to what you might think, actually makes it easier to stay awake! In another repeat from 2015, we started late and connecting to the projector proved problematic throughout the day. This time we had a selection of shiny MacBooks with their new-fangled USB-C connectors to thank!

The great thing about this conference is its independence which means that the sessions during the day covered the complete gamut of container technologies. Here’s a quick run down of the day:

  • Craig Box from Google kicked the day off. His session was billed as covering Kubernetes 1.3 but, as he pointed out, that was old hat with 1.4 due to release within a week. As such, he spent much of the pitch talking about what was coming up. To me it was a reminder to have a play with deploying various WebSphere topologies with Pet Sets.
  • Next up was Ben Firshman, repeating his serverless app talk from DockerCon. I keep meaning to ask whether he knows that OpenWhisk supports Docker containers as actions.
  • Michael Hausenblas from Mesosphere came after the break. He was talking about DRAX, his chaos testing tool for DC/OS.
  • Michael was following by Nishant Totla, giving his first conference presentation. He’s an engineer at Docker working on swarmkit, with orchestration in Docker 1.12 being the subject of his presentation.
  • Mark Shuttleworth of Canonical fame had the last session of the morning. He was talking about snaps for application packaging, particularly in the context of IoT devices.
  • Once the long lunch queue had finally subsided there was a series of lightning talks but, standing about three meters from the speakers, I still couldn’t hear half of what was said against the background. I’ll have to wait for the replays.
  • After lunch, Jonathan Boulle from CoreOS talked about the rkt container runtime and, in particular, the work that has been done to integrate it in to Kubernetes. Undoubtedly factoring the Docker-specifics out of Kubernetes has been beneficial to the project. It remains to be seen whether rkt overtakes Docker as the runtime of choice.
  • George Lestaris (now working on Garden for Pivotal) was talking about the project to use the CernVM File System as the backing for a container layered file system. Consider what if the large proportion of the content of many images that is never touched by the running process was never pulled across the file system?
  • Liz Rice had borrowed Julz Friedman’s pitch on building containers from scratch with Go. It was interesting to compare Liz’s style of “oh look – what would happen if I tried this?” versus Julz’s “let me show you my skills”!
  • Gareth Robertson then took to the stage briefly to plug RC1 for Label Schema which seeks to standardise a base set of Docker image labels.
  • After another break, Ed Robinson from Reevoo gave an entertaining pitch on the Træf?k reverse proxy. He talked about cheese a little bit too much though as this was point a mouse started to repeatedly traverse the flooring in front of me!
  • Chris Van Tuin from Red Hat gave an OpenShift pitch, lightly disguised as a presentation on container security.
  • Dustin Kirkland, another Canonical employee was talking about LXD and HPC. My attention started to drift at this point as watching the activities of the mouse proved more entertaining!
  • Docker Captain Alex Ellis rounded off the day with a Swarm/Raspberry Pi/IoT demo. You can’t beat a few flashing lights to please the audience!

Everything was being recorded so keep checking back on the conference YouTube channel for any sessions that peak your interest.

Summer Holidays: Act Three

Monday, September 5th, 2016

This final instalment is mostly taken up with our actual summer holiday. Taking a holiday in Britain at the end of August can sometimes be testing the definition of summer and, as we set off for Pembrokeshire, we were heading in to gale force winds. To be fair, this kept the roads fairly empty and, when we arrived at Broad Haven, meant there were some impressive waves breaking against the sea wall. We were staying in a ‘lodge’ at the same place we’d stayed six years earlier when Duncan would have been about 9 months old.

St David's CathedralSurfBy the following day, the wind had died down enough that Christine and the children could test out their new wetsuits body boarding in the sea. Unfortunately the rain returned before too long and we had to test out the selection of board games in the lodge. Things weren’t much better the following day and we tested out the swimming pool in Haverfordwest before taking a trip to St David’s for a look round the cathedral.

Pembroke CastleEmma body boardingThe sun finally made itself felt after that and we spent two pleasant days at the beach. Duncan has thankfully learnt not to eat sand in the intervening years! On another day we visited the privately owned Pembroke Castle which was a trip down memory lane for me having been there during a junior school trip to Tenby. We fell in on a guided tour where there was a good selection of gruesome stories to entertain the children. It was also, slightly randomly, circus skills day, and the children greatly enjoyed the Punch and Judy show.

Marloes PeninsulaSealsWe also revisited Martin’s Haven where the martins are still in residence in the toilets! We debated a trip to Skomer Island but it was too late in the year to see puffins. Instead we just wandered the cliff top path, looking down on the seals and their newborn pups below.

Llyn IdwalAs with our last trip to Pembrokeshire, it was followed by a drive up to north Wales. In an unfortunate reoccurrence, Emma was once again car sick on that journey. We stayed a couple of nights in Caernarfon to be close to Christine’s cousin and extended family who were staying on Anglesey. We took the children for a walk round Llyn Idwal which was unfortunately shrouded in damp mist. Christine and her cousin did a run/walk up to the Glyders and such was the visibility that they managed to descend on the wrong side of Tryfan!

Dave in the seaNewboroughIn contrast, we had glorious sunshine for the following day’s visit to Newborough Sands, scene of the British Orienteering Champs in 1995. While the others set off along the beach to the island (at least it’s an island at high tides) I had a run round the 10K+ Commonwealth Trail Champs route which is signposted.

We relocated to Bryn Gwynant Youth Hostel for the next couple of nights but met up with Cath and family again at Pen-y-Pass for an assault on Snowdon. Thankfully, unlike our last Snowdontrip along the Miner’s Track, no running buggies or baby carriers were required and this time the children made it all the way to the summit of Snowdon. Unfortunately the cloud never lifted as forecast and it was pretty miserable on top, not helped by the café being closed and Emma was heartbroken that she wouldn’t be able to spend any money in the shop! We descended back down the tourist track in to Llanberis for the traditional refuelling at Pete’s Eats.

Wilderhope ManorChristine had a grant interview in Swindon on the Thursday so we departed Wales and spent a night in the rather grand YHA Wilderhope Manor on Wenlock Edge. The stay was even more grand for the fact that our ‘en-suite room’ turned out to be the bridal suite! The mere presence of a bridal suite is a good indication of why we have never been able to book a room here at the weekend when orienteering in Shropshire.

Whilst Christine attended her interview, we amused ourselves at the nearby STEAM Museum of the Great Western Railway. It was billed as being an excellent way to pass STEAMa few hours and so it proved to be. There were a relatively small number of locomotives on display but this meant there was plenty of space to stand back and appreciate them. There were also lots of diversion for the children which meant that I could actually read some of the material on display. I hadn’t appreciated the extent to which Swindon owed its existence, or at least size, to the presence of the railway.

Paultons ParkThat brought us back home but, with Christine working a weekend open day at the University, I still had some child-minding to do and we decided to tick one more item of the children’s bucket list for the summer: a return trip to Paultons Park. The answer I posed at the end of my last blog post on this subject was 5 years, although Emma has managed a trip there with school in the interim. The children’s tastes have certainly matured and we only had one ride in Peppa Pig World (although this was possibly my worst with Duncan attempting to spin our cabin as fast as he could!). Thankfully the queues are somewhat shorter in other parts of the park, including the new rides in the Lost Kingdom. Emma demurred at some of the rides but this only spurred Duncan on and sadly he was the one who still needs to be accompanied by an adult on many of them! In the end, Emma caved in and joined us on everything. The only ride we didn’t do (although Duncan was definitely eyeing it up) was the Edge.

Summer Holidays: Act Two

Saturday, August 20th, 2016

Dyrham ParkWe returned to pick the children up from Monmouth the following weekend, stopping off at Dyrham Park on the way back. Quite apart from it being a convenient place just off the M4 to stop for lunch, the children had last been there when the roof was in the process of being replaced following a fire and were keen to see how it now looked. The scaffolding that they had previously ascended to look down on the roof was all gone and, at least externally, the property was once again looking in fine form.

Emma runningChristine and Duncan runningOn the Sunday Christine took Emma along to the Junior parkrun in Southampton. Emma must have enjoyed it as she was persuaded out their again the following weekend where, despite getting a lower place, she managed to beat her previous time. Duncan also decided that he would go round this time with Christine. Thankfully he managed to maintain a respectful distance behind is older sister! All of this now means that I’m the only one in the family never to have run a parkrun despite being the first to register for a barcode! Duncan climbingI ran home whilst Christine and the children went to investigate what turned out to be an open day at Woodmill Outdoor Activities Centre. This gave them a chance to test out their climbing skills yet again.

Spinnaker TowerChristine and I took it in turns to mind the children the following week. I took them down to Portsmouth Historic Dockyard on the train for one day. We went up the Spinnaker Tower first which, to be honest, was a bit of a let down. Other than a view out over the Solent to the Isle of Wight there really just isn’t much to see. There was quite some queue to get in to the dockyard itself but we can return on our ticket at any point during the next year and won’t have to suffer the queue again. After a stop for lunch we headed over to see the Mary Rose in her new dry state. Mary RoseIt’s very well done with the skeletal remains of the one half of the ship on one side and the retrieved contents was laid out on the other in decks as it would have been found on the ship. Unfortunately I think the sheer age of the ship and the remarkableness of the fact that any of it actually remains today was lost on the children.

HMS WarriorEmma had been to see HMS Victory before with school and Duncan will go this year so we then took a look around HMS Warrior. Despite the fact that pretty much only the shell is original, the children still got much more out of out this. Later in the week we took a trip out to Mottisfont to check out the Beatrix Potter trail. On what was a sunny day, it was absolutely heaving and I think the children may finally be starting to outgrow some of the activities (although not the ice cream at the end!).

Howzat!Excitement for the following weekend consisted of a trip to the Ageas Bowl, this time to watch women’s Twenty20 cricket. It was every bit as exciting as the men’s game and the home side of the brightly clad Southern Vipers played a convincing victory over Loughborough Lightning. It was then back to work for Christine and I with the children attending summer camp at their school for the next week.

Summer Holidays: Act One

Saturday, August 6th, 2016

The Summer school holidays began with an event of mixed emotions as I coaxed the ailing (flightless and quite possibly sightless) juvenile crow that had occupied our back garden for the past month or so in to a cardboard box. Whilst I was out at work, Christine and the children deposited it at the local vets. What its fate was there we don’t know. It does, however, mean that we are no longer woken early by its cries for food and that we can hang the washing out unmolested!

Go ApeThe fun started in earnest at the weekend when we took the children to Junior Go Ape at Itchen Valley Country Park. Emma has been angling to go for some time now. They are both old enough/tall enough to be able to go round on their own so Christine and I could watch from ground level. Go APeUnlike the adult version, you’re attached permanently to a wire from start to finish which places some limits on what you can do but means that there is no chance of them falling and there is no faffing around with clipping and unclipping the whole time. They also get to go round the two loops on offer as many times as they like in the hour slot which, in Emma and Duncan’s case, was lots!

ExburyExburyExburyI looked after the children for one day the next week and we took a trip down to Exbury Gardens. The gardens are probably not at their best at this time of year but I was still taken aback by quite how deserted the place was. ExburyWe went round the family trail which was educational for all and then took a ride on the train. We’ll have to return again in the autumn or spring.

Camping PodFor the second half of the week, Christine and the children were checked in to a ‘camping pod‘ at South Downs Youth Hostel which they all seemed to enjoy, even if they did spend rather a lot of time waiting for Southern Rail.

Bristol ZooBristol ZooThe children were staying with Christine’s parents for the second full week of the holiday. We drove them up and met up with Christine’s brother and family at Bristol Zoo on the way there.Climbing It’s amazing how much they pack in to such a small area and it’s certainly the equal of Marwell.

Whilst away, the children enjoyed some more high altitude action with a return trip to the climbing wall at Llangorse Lake.

Book Review: Mastering Docker and Monitoring Docker

Sunday, June 19th, 2016

Mastering DockerThe films on my flight to the US this week weren’t much to write home about so I ploughed through some of my Safari Queue backlog. Two of the Docker related books on my queue were from Packt Publishing.

The first was Mastering Docker by Scott Gallagher which I’ll confess that I only skim read. Parts of the book were already showing their age even though it was only published in December 2015 but that’s inevitable in such a fast-moving area. More problematic was that I found myself disagreeing with the author so much in just the first couple of chapters that I couldn’t believe anything I read after that. By way of just one example: the author asserts that commands are chained together in Dockerfiles to speed build times and seems to suggest that items added in one layer may be removed in another to keep the image size down. The structure of the book is also poor with material repeated throughout. It certainly doesn’t contain the depth to provide mastery in anything!

Monitoring DockerThe second book from the same stable was Monitoring Docker by Russ McKendrick. Maybe it’s just because this is an area that I know less about but I got on better with this book. It covers use of the Docker API (top, stats and logs), cAdvisor and aggregation with Prometheus. The author covers Zabbix in some detail which, I must confess, is a tool that I’d never even heard of, before a gushing endorsement for Sysdig. The book provides an overview of SaaS options (Sysdig Cloud, Datadog and New Relic) before closing with a detailed walk-through of setting up a containerized ELK stack for log aggregation. There are certainly options that the book didn’t cover but I felt it provided good coverage of the considerations to allow the reader to make informed decisions about what they should look for in a monitoring solution.

I also took a look at the early release of Kelsey Hightower’s Kubernetes: Up and Running from O’Reilly. It shows promise but, as it’s not due for release until October, there wasn’t enough content there yet to give a review.

WebSphere Liberty and IBM Containers: Part 3

Saturday, June 18th, 2016

Scaling up

In the first two posts in this series we covered the basics of starting a WebSphere Liberty on IBM Containers via the browser and then using the command line to deploy an application.

We’ve already seen some of the value-add that comes out of the box when running under IBM Containers. For example, at no point have we needed to be concerned with the underlying infrastructure on which the containers are running (beyond selecting a region). When we created an image it was scanned automatically for vulnerabilities. Each container was allocated its own private IP address accessible from any other container running in the same space – no need to set up and configure overlay networking here. We had control over whether we also wanted to assign a public IP and, if so, what ports should be exposed there. We also had easy access to metrics and standard out/error from the container.

So far we’ve only deployed a single container though. What happens when we hit the big time and need to scale up our application to meet demand? When we created our first container via the UI, you may remember that the Single option was selected at the top. Let’s go back and try out the Scalable alternative. From the catalog, navigate through Compute and Containers (remember that these instructions are based on the new Bluemix console). Select our existing demo image. Next, select the Scalable button at the top and give the container group a name. By default you’ll see that our group will contain two instances.

Rather than having a single IP associated with a container, this time we are asked to give a host name (on the mybluemix.net domain by default) for the group. Requests arriving at this host name will be load-balanced across the container instances in the group (reusing the gorouter from Cloud Foundry). One nice bonus of this approach is that it doesn’t eat in to our quota of public IPs! As the host name needs to be unique within the domain, I tend to include my initials as you’ll see in the screenshot below. Select 9080 as the HTTP port and then click Create.

Container group creation

Once the containers have started, the dashboard should show that you have two instances running:

Running instances

Right-click on the route shown in the Group details section and open it in a new tab. This should take you to a Liberty welcome page and, if you add the myLibertyApp context root, you should be able to see the application again. If you hit refresh a few times, although you won’t be able to tell with this application, your requests will be load-balanced across the two instances. If you return to the dashboard and switch to the Monitoring and Logs tab you can switch between the output for the instances and should, for example, be able to see the spike in network usage on the two containers when you made the requests.

If you return to the Overview tab you will see that there are plus and minus symbols either side of the current number of instances. These can be used to manually scale the number of instances. Click the + icon, click Save, and watch the creation of a new container in the group.

Manual scaling is all very well but it would be better if the number of instances scaled automatically up and down as required. If you’re deploying your containers in the London region then you’ll notice an extra tab at the top of the dashboard labelled Auto-Scaling. It’s only available in the London region at the moment because the service is still in beta (and so things may change a bit from what I’m describing here). Having selected this tab, click the plus icon labelled Create policy. Give the policy a name default and set the minimum and maximum instance values to 1 and 3. Add two CPU usage rules to scale up and down the number of instances as shown in the following diagram and then hit Create. Finally, select Attach to activate the policy for this scaling group.

Auto-Scaling

If you click the Auto-Scaling History tab you should see that a scaling action has taken place. We originally scaled up manually to 3 instances but, as the CPU usage is below our 60% limit, the number gets scaled down by one. If you wait another 5 minutes (the cool down period we specified), then you’ll see it get scaled down again to our minimum of 1.

Scaling history

And that concludes our tour of the scaling options in IBM Containers!