Multi-Stage Docker Build

May 5th, 2017

Docker 17.05 enabled the ability to perform multiple build stages in a single Dockerfile, copying files between them. This brings to regular Docker build a capability that I’ve previously talked about in the context of Rocker, and something that’s of particular use in a compiled language like Java. Let’s see what it would look like in the context of the WebSphere Liberty ferret sample.

The original Dockerfile looks as follows:

We can see that it assumes that the application has already been built and just pulls in the WAR file, in this case from Maven Central. With a multi-stage build we can perform the build of the application and the build of the image in a single Dockerfile:

The first line uses the Maven on-build image to build the application using the source in the same directory as the Dockerfile. Although the stages are given an index by default, naming them using the AS keyword makes the file much more readable. Further down in the Dockerfile we can see that the COPY command takes the built WAR file from the first stage and copies it into the Liberty dropins directory as before. The important thing about all of this is that the final image doesn’t end up with the application source in it, or Maven, or an SDK – just the artifacts that are needed at runtime – thereby keeping down the final image size.

Posted in Docker, Technology, WebSphere Application Server, Work | Comments Off on Multi-Stage Docker Build

Lite Weekend of Running

May 2nd, 2017

 We had a fun Bank Holiday weekend in South Wales competing in the OMM Lite. Christine and I were entered on the Long Score with her Mum having offered to mind the children at the event centre in Cwm-du, nestled in the Black Mountains. The Lite is significant as the event is quite different from the OMM itself. Firstly, as with the Capricorn, you return to the event centre on Saturday so there is no need to carry camping equipment or food. Secondly, the weather at this time of year was considerably better than your average OMM. Lastly, the event was restricted to using rights of way only. This latter point makes a major difference: the navigation was simple, the course was much more runnable, and lastly, it meant that the route choice options were much more limited.

In the seven hours on Saturday, we ended up running 53km – considerably more than either of us had been expecting. This included an ascent of Waun Fach. Unfortunately, with Pen Cerig-calch effectively out-of-bounds it also meant that, with two hours to go, there was nothing for it but a long run back round the lanes and connecting footpaths, picking up a few checkpoints on the way. We finished the first day in second place. The winners were miles ahead of us (rumour has it that they ran a further 10km) but there was another mixed pair just behind us who we knew had been running faster than us.

The map had all of the available checkpoints for the weekend marked on it and, although it was only at the start of each day that you discovered the controls that were open and how much they were worth, there was still plenty of opportunity for route planning on Saturday night. In the end, we only made one small tweak to the route I had chosen, taking in Mynydd Troed at the beginning and returning to Mynydd Llangorse, where Christine’s Dad was stationed, towards the finish. With blisters from her new Inov-8s, Christine chose to wear road shoes on Sunday and wasn’t significantly disadvantaged. In contrast, my new Inov-8 Talon 212s didn’t give me any trouble despite only having worn them for half an hour before the event.

Rather embarrassingly, our attempt to avoid the Brecons over Easter backfired completely as there was a control within 50 metres of where we had parked the car at Llangorse Lake! It didn’t give us any advantage though with no option for canoeing across the lake! Christine’s knee started to give her some trouble with 1.5 of the 5 hours still to go but she soldiered on and, as we had time, was even persuaded to take in an extra control at the end. This brought our distance for the second day to nearly 35km, gave us a win for day 2 and confirmed our place as second overall and first mixed-pair.

Overall, a fun weekend although, as I say, quite different from the OMM. It was certainly family friendly with the children enjoying the organised walks and the mountain bike skills course laid on at the event centre. The Lite format has another couple of events in the south with the Chilterns and Surrey Hills but, even with the requirement to stick to the paths, I don’t think they can compete with being out on the open fells.

 

Posted in Family, Orienteering, Running, Travel | Comments Off on Lite Weekend of Running

Easter Part II

April 21st, 2017

After the JK we headed over to Monmouth to spend a few days with Christine’s parents. The Tuesday was a lovely day and we stopped off at Westonbirt Arboretum on the way. Although a little pricey for a Forestry Commission venue there is plenty to see and we spent more time there than we had originally intended. As with, I suspect, most first-time visitors, we set off along the Treetop Walkway that takes you 13 metres up into the trees. As well as being a beautiful structure, it also has a mine of interesting information (most of which I’m afraid I immediately forgot).

After a picnic lunch, we continued around the Spring Trail which took us through the Cherry Glade to the Silk Wood barn where there was a selection of Easter activities on offer (and an opportunity to stock up on yet more chocolate!). After briefly checking out the play area next to the café (Emma was stretching the declared age range a little), we continued on into the Old Arboretum. As well as dog free zone, it also seemed to be largely toddler free which made for a much more peaceful walk than the first half.

On Wednesday we headed over to Llangorse Lake to meet up with Christine’s cousin, her family, and her extended family-in-law. We started with a walk around the lake where we temporarily lost a pair of shoes in the mud where the field had been under water just a week or two ago. Having returned to the cars for lunch, we then took to the water on an assortment of different boats (rowing plus one- and two-seater and Canadian canoes).

Christine headed off to a conference in Birmingham of the next two days and the rest of us had a quiet day around Monmouth on Thursday. We still managed to visit the Museum (an eclectic mix of Nelson and Rockfield Studios memorabilia) and Shire Hall both of which were firsts for me despite 20+ years of visiting the town. The latter is full of gruesome stories from its days as Assize Court.

On Friday I returned home with the children but we stopped off on in Gloucester on the way back with Christine’s parents. Emma hadn’t got her climbing fix at Llangorse so the kids went bouldering at The Warehouse. We then had our lunch down by the quays and wandered around the waterfront until our parking ran out.

Just two more days until everyone is back at work!

Posted in Family, Sport, Travel | Comments Off on Easter Part II

JK 2017

April 17th, 2017

We spent most of the Easter weekend south of London orienteering at the JK. We didn’t go to the sprint on Friday (quite frankly, it didn’t seem worth the high entry fee) so our first event was the medium race on Saturday on Ambersham Common. Christine went out first and had a respectable run finishing third on W40. I amused the children with the string course before we walked Emma to the start for her first W10B course. Unfortunately she took a wrong turn and missed out a control, something that Duncan didn’t repeat when he then ran the white course with Christine in tow (it was the same course as Emma’s). I had a scrappy start to my run, wasting a good couple of minutes on #7. Roger Goddard gave me a tow for a while until Geoff Ellis took him away. 12th place was set to become a recurring theme for the weekend.

Part of the draw for the weekend was a chance to catch up with friends over from the Czech Republic and we had a nice meal on the Saturday night. Sunday was the classic distance race and I had a long trek over to my start in St Leonard’s Forest. No major blunders this time but I don’t have the speed in the rough terrain as the course wound its way back on to the Holmbush map and I finished… 11th but still 12th over the two days! Emma took another wrong turn on her course but recovered successfully this time. She was still beaten by Duncan though who went out on his first course unaccompanied. Christine had another successful day in the forest and we stayed to watch her collect her 3rd prize.

There was just a string course for the children on Monday and Christine and I made up either end of a Men’s Short team. I was off first and was pleased to discover there were only seven finishers in front of me with second place under 90 seconds ahead. Dan put in a sterling effort for his first relay, holding on to 8th place. After Christine’s run we finished a respectable… 12th.

It was a fun weekend of orienteering and, probably most importantly, the children seemed to have enjoyed themselves and are looking forward to our summer orienteering holiday.

Posted in Family, Orienteering, Travel | Comments Off on JK 2017

Introducing Microservice Builder

March 27th, 2017

When the frequency of blog posts drops on this site it generally has two causes: I’m busy and/or I’m working on something that’s IBM Confidential. Both of these have been true over the past six months or so whilst I’ve been working on something we’re calling Microservice Builder. A public beta was announced in the run up to InterConnect which went live on the 24th which means that I can now come up for air and say a little about the work we’ve done so far.

Although not limited to Java deployments, Microservice Builder pulls together multiple strands of work that we’ve been doing in the WebSphere space. First, there is the work that is being done in the MicroProfile community to define a set of standard APIs for building microservices in Java. Initially, this took a set of existing Java EE technologies (JAX-RS, CDI and JSON-P) but now additional APIs are being defined. You can start to see the results of this work in the Liberty March beta where there are new features for injecting environmental configuration and utilizing fault tolerance patterns such as timeout, bulkhead and circuit breaker.

Another area where we’ve sought to improve the developer experience is by providing a fast-path to creating new projects. The Liberty App Accelerator has been around for some time now, allowing you to generate Java projects quickly through a web UI. We’ve taken this idea and extended it to cover Swift and Node.js. This can be achieved either through a web UI or through a new plugin to the Bluemix CLI. (Note that generated projects do not need to be deployed to Bluemix.) The plugin goes beyond just generating projects and allows you to build and run them locally using containers. This means that the developer no longer needs to have the prerequisites (e.g. Java, Maven and Liberty) installed locally.

For a runtime environment, we believe containers are a good fit for microservices and in the first instance we’re focusing on Kubernetes. That could be the newly announced Kubernetes in IBM Containers or it could be on-premises with IBM Spectrum Conductor for Containers. On top of Kubernetes, Microservice Builder adds a lightweight fabric, installed as a Helm chart, that simplifies deployment of Liberty-based services. Specifically, in this first release it generates key and trust stores to facilitate inter-service communication. It also configures an ELK (Elasticsearch-Logstash-Kibana) stack to receive and display information including trace, FFDC, garbage collection and HTTP access logs from the Liberty logstashCollector-1.0 feature.

The final strand of Microservice Builder ties together the development and runtime environments via a Jenkins based pipeline. Once again, this is installed as a Helm chart, and is configured to automatically pick up projects from a GitHub or GitHub Enterprise organization. For a Java application, the pipeline will build and test using Maven, before creating a Docker image and pushing it to a registry. The Docker image is then deployed to a Kubernetes cluster using either the same or a separate pipeline.

To show all of this in action, we have taken the sample conference application from the MicroProfile community and broken it apart in to separate projects to deploy using Microservice Builder. Just follow the docs to recreate it in either your local minikube environment or with Spectrum Conductor for Containers.

Posted in Cloud, Docker, WebSphere Application Server, Work | 1 Comment »

Presentations from IBM InterConnect 2017

March 26th, 2017

I’m finally back home after what feels like a very long week in Las Vegas at IBM’s InterConnect conference. I promised that I’d post my presentations on SlideShare and I’ll add a few comments here on how each session went.

After an Inner Circle session on Sunday, my first public session of the week was an introduction to containers with WebSphere traditional. This played to a full room which suggests that there is significant interest in the use of containers for existing workloads. Indeed, that was the point of the second half of the session, to describe scenarios where it may make sense to use containers with traditional WebSphere. That’s not to say that it always does and, during one-to-one sessions during the week, I found myself repeatedly cautioning customers against rushing in to the use of containers, particularly with ND, just for the sake of it.

https://www.slideshare.net/davidcurrie/how-to-containerize-websphere-application-server-traditional-and-why-you-might-want-to

My next session covered our new announcement around Microservice Builder. I’ll not say more here as I’ll cover this in a separate post.

https://www.slideshare.net/davidcurrie/microservice-builder-a-microservice-devops-pipeline-for-rapid-delivery-and-promotion

Unfortunately, I didn’t get to deliver this session on Liberty and IBM Containers as it clashed with another that I was presenting. As touched on briefly in this presentation, one of the other announcements at the conference was for support for Kubernetes in IBM Containers. There was lots of excitement around this and I urge you to go and check it out for yourself.

https://www.slideshare.net/davidcurrie/websphere-liberty-and-ibm-containers-the-perfect-combination-for-java-microservices

On Wednesday I had a joint presentation with Brian Paskin looking at options for scalability with Liberty and containers. This was very much Brian’s presentation though so I shan’t post it here. There was an accompanying lab in the afternoon that looked at Liberty collectives and at IBM Containers.

My last session of the week was looking at some of the options when choosing a container orchestration platform: from Liberty collectives, through Swarm and Docker Datacenter, and Kubernetes with IBM Spectrum Conductor for Containers and IBM Containers. Many customers I spoke to this week were looking for a single definitive answer here but my response for now is still very much “it depends”.

https://www.slideshare.net/davidcurrie/choosing-a-container-platform-for-your-websphere-applications

Posted in Cloud, Docker, Travel, WebSphere Application Server, Work | Comments Off on Presentations from IBM InterConnect 2017

Find me at IBM InterConnect 2017

March 18th, 2017

I’m going to be at IBM’s InterConnect conference this coming week. If you’re going to be there too, there’s a quick run-down of the sessions I’ll be presenting below. The astute will notice that, due to a scheduling snafu, I’m supposed to be presenting two sessions at the same time on Tuesday. If you go to the Liberty/ IBM Containers session then I’m afraid you’ll have to make do with Tom – be kind to him!

If you want to chat about any combination of microservices, containers and WebSphere, you can find me on the microservices ped in the WebSphere area of the expo hall from 5-7:30pm on Tuesday and again from 3-5pm on Wednesday. I’ll be kicking off the latter with a live demo of Microservices Builder, of which more in another post. For Inner Circle customers, I’ll also be talking about this topic at 11am on Sunday.

HAJ-5451 : How to Containerize WebSphere Application Server Traditional, and Why You Might Want To
Date/Time : Mon, 20-Mar, 11:15 AM-12:00 PM
Location : Mandalay Bay South, Level 2 – Surf D
Presenter(s) : David Currie, IBM

BMC-7014 : Roundtable Discussion on Building Java Microservices with WebSphere Liberty
Date/Time : Mon, 20-Mar, 02:00 PM-02:45 PM
Location : Mandalay Bay North, Level 0 – Tropics A
Presenter(s) : Alasdair Nottingham, IBM; David Currie, IBM

BMC-7085 : Meet the Expert on IBM WebSphere Application Server Liberty on Docker
Date/Time : Tue, 21-Mar, 02:30 PM-03:15 PM
Location : Concourse, Bayside B, Level 1 – Meet the Experts Forum # 1
Presenter(s) : David Currie, IBM; Tom Banks, IBM

HAM-5526 : IBM Microservice Builder: A Microservice DevOps Pipeline for Rapid Delivery and Promotion
Date/Time : Tue, 21-Mar, 03:45 PM-04:30 PM
Location : Mandalay Bay North, Level 0 – Islander F
Presenter(s) : David Currie, IBM; Jeremy Hughes, IBM

BMC-5983 : WebSphere Liberty and IBM Containers: The Perfect Combination for Java Microservices
Date/Time : Tue, 21-Mar, 03:45 PM-04:30 PM
Location : Mandalay Bay North, Level 0 – South Pacific A
Presenter(s) : David Currie, IBM; Tom Banks, IBM

BMC-7014 : Roundtable Discussion on Building Java Microservices with WebSphere Liberty
Date/Time : Wed, 22-Mar, 08:00 AM-08:45 AM
Location : Mandalay Bay North, Level 0 – Tropics A
Presenter(s) : Alasdair Nottingham, IBM; David Currie, IBM

BMC-2714 : Utilizing WebSphere Application Server Liberty in Docker Containers for Scalability
Date/Time : Wed, 22-Mar, 10:15 AM-11:00 AM
Location : Mandalay Bay North, Level 0 – South Pacific A
Presenter(s) : Brian S. Paskin, IBM; David Currie, IBM

HAJ-2718 : Utilizing IBM WebSphere Liberty in Docker Containers for Scalability (Lab)
Date/Time : Wed, 22-Mar, 01:00 PM-02:45 PM
Location : Mandalay Bay South, Level 3 – South Seas H
Presenter(s) : Brian S. Paskin, IBM; David Currie, IBM

BAS-5901 : Choosing a Container Platform for Your WebSphere Applications
Date/Time : Thu, 23-Mar, 10:30 AM-11:15 AM
Location : Mandalay Bay North, Level 0 – South Pacific A
Presenter(s) : David Currie, IBM; Tom Banks, IBM

Posted in Cloud, Docker, Travel, WebSphere Application Server, Work | Comments Off on Find me at IBM InterConnect 2017

How many processors?

January 31st, 2017

Reading Daniel Bryant’s O’Reilly publication Containerizing Continuous Delivery in Java reminded me of the challenge of determining how processors are available to you when running in a container. In the case of Java, a call to Runtime.getRuntime().availableProcessors() should show this all important information. A quick check reveals that, when called in an unconstrained container, this correctly returns the number of cores on my physical hardware (Docker on Linux) or assigned to the VM containing the Docker Engine (Docker Toolbox or Docker for Windows/Mac). If I used the --cpuset-cpus option on docker run to constrain the cores available to the container then this is also correctly reflected in the value returned. The difficulty arises when access to those CPUs is constrained in other ways.

Take, for example, the new --cpus option in Docker 1.13. Setting this to two on a four-way box, I still get four back from a call to availableProcessors() and rightly so: there are four processors and I may get simultaneous access to all four of them even if the cgroup is then going to make sure that I don’t get that access for more than half of the time. Another potential constraint is a highly multi-tenant environment. If I deploy my test application to Bluemix it tells me that there are 48 processors. That’s great but I’m pretty sure I’m not going to get exclusive access to all of those!

One example we’ve seen where this becomes a real problem is in native memory usage. By default, WebSphere Liberty uses the number of available processors to decide the number of parallel threads it should support. Each of those executors utilises a thread and each of those threads takes up space in native memory. In a containerized environment where total memory is typically constrained (Bluemix containers are sold by the GB/hour) and some generic heuristic is often used to determine the heap size to allocate to the JVM, that can lead to memory exhaustion. That’s why you’ll see a GitHub issue from my colleague Erin that, among other things, proposes setting hard-coding a maximum on the number of threads for the executor service in our Docker images.

Posted in Docker, WebSphere Application Server | Comments Off on How many processors?

« Older Entries
Newer Entries »