Archive for the ‘Cloud’ Category

Jenkins X

Wednesday, September 19th, 2018

Having started to get some rhythm back in to the publishing of my personal blog posts, I thought it was about time that I started posting some technical content again too. I’m having to do lots of new learning at the moment and, if nothing else, writing about it makes sure that I’ve understood it and helps me remember at least some of what I’ve learnt. As before, these posts in no way indicate the position of my employer nor, in general, should you read in to them anything about technical direction. On the whole, they are just about topics that I’ve found sufficiently interesting to write a little about. There is, I have found, no knowing what will be of interest to other people (my all-time top post relates to Remote Desktop!). From an entirely selfish perspective, I don’t care if anyone reads what I write as it’s largely the writing that gives me value!

Having said all the above, the subject of this post is Jenkins X which is very much in the domain of my new employer! When it was announced back in March, I have to admit that I was somewhat sceptical. It was clearly aiming at much the same space that we were with the dev-ops part of Microclimate. My view wasn’t helped by the fact that I couldn’t actually get it to run. It did (and still does) run best out on public cloud but I used up my free quotas on AWS and GCP a long time ago. I tried to run it on minikube and failed. It was also developed by the team behind Fabric8 which, although it showed lots of promise, was never incorporated in to any of RedHat’s commercial offerings. The same was not set to be true of Jenkins X and, six months later, my new employer has just announced that it now forms part of the CloudBees Core offering under the name of Kube CD. I’ll save details of that commercial offering for another post and restrict myself to talking about the open source Jenkins X project here.

So what exactly is Jenkins X? It enables Continuous Integration and Continuous Deployment, of applications on Kubernetes. It happens to use Jenkins as the engine to perform those actions but, at least at a first pass, that is immaterial. Around that Jenkins is wrapped lots of Kubernetes-native goodness and, most importantly, a CLI by the name of jx. Thankfully this time around the minikube experience worked for me just fine and getting up and running was as simple as:

I have to say that I’m not a big fan of ‘verb followed by noun’ when it comes to CLI arguments as, although perhaps more readable, it makes end-user discovery harder (jx create tells me about a whole long list of largely unrelated things) but thankfully just typing jx gives a reasonable overview of the main options. Beware though that the CLI is heavily overloaded: it’s not only used for initialisation, but also subsequent actions performed by the developer, and those performed by the pipeline.

Perhaps the quickest way to demonstrate the capabilities is to then use a quick start:

This allows you to select a technology stack (everything from Android to Vert.X via Rust, Rails and React!) which lays down a skeleton application on disk. It then uses Draft to add a Dockerfile and Helm chart(s). It doesn’t stop there though. It will then help you create a repo on GitHub, check your code in, set up a multi-branch pipeline on the Jenkins instance it provisioned, and set up the web hook to trigger Jenkins on subsequent updates. (Webhooks don’t tend to work too well unless your minikube is internet facing but, given a bit more time, polling does the job eventually.) The default pipeline (defined by a Jenkinsfile in your application repository) uses Skaffold to build the application Docker image and push to a registry. The Helm chart is published to the provided instance of ChartMuseum.

Jenkins X follows the GitOps model promulgated by Alexis Richardson and the team at Weaveworks. By default, it sets up two GitHub repositories that map to staging and production namespaces in the Kubernetes cluster. Additional environments can easily be defined via, you guessed it, jx create environment. These repositories make good use of ‘umbrella’ Helm charts to deploy specific versions of each of the application charts. By default, the master branch is automatically deployed to the staging environment but promotion to production is performed manually, for example:

There is also the concept of a preview environment. Typically created for reviewing a pull request, they can also be created manually via the CLI. These allow a specific version of the application to be accessed in a temporary namespace created just for that purpose. All of the Jenkins X configuration (environments, releases, …) are represented in the Kubernetes way: as Custom Resource Definitions.

There’s plenty more to say about Jenkins X but I’ll save that for another post on another day. Hopefully this has given you enough of a flavour to encourage you to download the CLI and give it a try for yourself.

Introducing Microservice Builder

Monday, March 27th, 2017

When the frequency of blog posts drops on this site it generally has two causes: I’m busy and/or I’m working on something that’s IBM Confidential. Both of these have been true over the past six months or so whilst I’ve been working on something we’re calling Microservice Builder. A public beta was announced in the run up to InterConnect which went live on the 24th which means that I can now come up for air and say a little about the work we’ve done so far.

Although not limited to Java deployments, Microservice Builder pulls together multiple strands of work that we’ve been doing in the WebSphere space. First, there is the work that is being done in the MicroProfile community to define a set of standard APIs for building microservices in Java. Initially, this took a set of existing Java EE technologies (JAX-RS, CDI and JSON-P) but now additional APIs are being defined. You can start to see the results of this work in the Liberty March beta where there are new features for injecting environmental configuration and utilizing fault tolerance patterns such as timeout, bulkhead and circuit breaker.

Another area where we’ve sought to improve the developer experience is by providing a fast-path to creating new projects. The Liberty App Accelerator has been around for some time now, allowing you to generate Java projects quickly through a web UI. We’ve taken this idea and extended it to cover Swift and Node.js. This can be achieved either through a web UI or through a new plugin to the Bluemix CLI. (Note that generated projects do not need to be deployed to Bluemix.) The plugin goes beyond just generating projects and allows you to build and run them locally using containers. This means that the developer no longer needs to have the prerequisites (e.g. Java, Maven and Liberty) installed locally.

For a runtime environment, we believe containers are a good fit for microservices and in the first instance we’re focusing on Kubernetes. That could be the newly announced Kubernetes in IBM Containers or it could be on-premises with IBM Spectrum Conductor for Containers. On top of Kubernetes, Microservice Builder adds a lightweight fabric, installed as a Helm chart, that simplifies deployment of Liberty-based services. Specifically, in this first release it generates key and trust stores to facilitate inter-service communication. It also configures an ELK (Elasticsearch-Logstash-Kibana) stack to receive and display information including trace, FFDC, garbage collection and HTTP access logs from the Liberty logstashCollector-1.0 feature.

The final strand of Microservice Builder ties together the development and runtime environments via a Jenkins based pipeline. Once again, this is installed as a Helm chart, and is configured to automatically pick up projects from a GitHub or GitHub Enterprise organization. For a Java application, the pipeline will build and test using Maven, before creating a Docker image and pushing it to a registry. The Docker image is then deployed to a Kubernetes cluster using either the same or a separate pipeline.

To show all of this in action, we have taken the sample conference application from the MicroProfile community and broken it apart in to separate projects to deploy using Microservice Builder. Just follow the docs to recreate it in either your local minikube environment or with Spectrum Conductor for Containers.

Presentations from IBM InterConnect 2017

Sunday, March 26th, 2017

I’m finally back home after what feels like a very long week in Las Vegas at IBM’s InterConnect conference. I promised that I’d post my presentations on SlideShare and I’ll add a few comments here on how each session went.

After an Inner Circle session on Sunday, my first public session of the week was an introduction to containers with WebSphere traditional. This played to a full room which suggests that there is significant interest in the use of containers for existing workloads. Indeed, that was the point of the second half of the session, to describe scenarios where it may make sense to use containers with traditional WebSphere. That’s not to say that it always does and, during one-to-one sessions during the week, I found myself repeatedly cautioning customers against rushing in to the use of containers, particularly with ND, just for the sake of it.

https://www.slideshare.net/davidcurrie/how-to-containerize-websphere-application-server-traditional-and-why-you-might-want-to

My next session covered our new announcement around Microservice Builder. I’ll not say more here as I’ll cover this in a separate post.

https://www.slideshare.net/davidcurrie/microservice-builder-a-microservice-devops-pipeline-for-rapid-delivery-and-promotion

Unfortunately, I didn’t get to deliver this session on Liberty and IBM Containers as it clashed with another that I was presenting. As touched on briefly in this presentation, one of the other announcements at the conference was for support for Kubernetes in IBM Containers. There was lots of excitement around this and I urge you to go and check it out for yourself.

https://www.slideshare.net/davidcurrie/websphere-liberty-and-ibm-containers-the-perfect-combination-for-java-microservices

On Wednesday I had a joint presentation with Brian Paskin looking at options for scalability with Liberty and containers. This was very much Brian’s presentation though so I shan’t post it here. There was an accompanying lab in the afternoon that looked at Liberty collectives and at IBM Containers.

My last session of the week was looking at some of the options when choosing a container orchestration platform: from Liberty collectives, through Swarm and Docker Datacenter, and Kubernetes with IBM Spectrum Conductor for Containers and IBM Containers. Many customers I spoke to this week were looking for a single definitive answer here but my response for now is still very much “it depends”.

https://www.slideshare.net/davidcurrie/choosing-a-container-platform-for-your-websphere-applications

Find me at IBM InterConnect 2017

Saturday, March 18th, 2017

I’m going to be at IBM’s InterConnect conference this coming week. If you’re going to be there too, there’s a quick run-down of the sessions I’ll be presenting below. The astute will notice that, due to a scheduling snafu, I’m supposed to be presenting two sessions at the same time on Tuesday. If you go to the Liberty/ IBM Containers session then I’m afraid you’ll have to make do with Tom – be kind to him!

If you want to chat about any combination of microservices, containers and WebSphere, you can find me on the microservices ped in the WebSphere area of the expo hall from 5-7:30pm on Tuesday and again from 3-5pm on Wednesday. I’ll be kicking off the latter with a live demo of Microservices Builder, of which more in another post. For Inner Circle customers, I’ll also be talking about this topic at 11am on Sunday.

HAJ-5451 : How to Containerize WebSphere Application Server Traditional, and Why You Might Want To
Date/Time : Mon, 20-Mar, 11:15 AM-12:00 PM
Location : Mandalay Bay South, Level 2 – Surf D
Presenter(s) : David Currie, IBM

BMC-7014 : Roundtable Discussion on Building Java Microservices with WebSphere Liberty
Date/Time : Mon, 20-Mar, 02:00 PM-02:45 PM
Location : Mandalay Bay North, Level 0 – Tropics A
Presenter(s) : Alasdair Nottingham, IBM; David Currie, IBM

BMC-7085 : Meet the Expert on IBM WebSphere Application Server Liberty on Docker
Date/Time : Tue, 21-Mar, 02:30 PM-03:15 PM
Location : Concourse, Bayside B, Level 1 – Meet the Experts Forum # 1
Presenter(s) : David Currie, IBM; Tom Banks, IBM

HAM-5526 : IBM Microservice Builder: A Microservice DevOps Pipeline for Rapid Delivery and Promotion
Date/Time : Tue, 21-Mar, 03:45 PM-04:30 PM
Location : Mandalay Bay North, Level 0 – Islander F
Presenter(s) : David Currie, IBM; Jeremy Hughes, IBM

BMC-5983 : WebSphere Liberty and IBM Containers: The Perfect Combination for Java Microservices
Date/Time : Tue, 21-Mar, 03:45 PM-04:30 PM
Location : Mandalay Bay North, Level 0 – South Pacific A
Presenter(s) : David Currie, IBM; Tom Banks, IBM

BMC-7014 : Roundtable Discussion on Building Java Microservices with WebSphere Liberty
Date/Time : Wed, 22-Mar, 08:00 AM-08:45 AM
Location : Mandalay Bay North, Level 0 – Tropics A
Presenter(s) : Alasdair Nottingham, IBM; David Currie, IBM

BMC-2714 : Utilizing WebSphere Application Server Liberty in Docker Containers for Scalability
Date/Time : Wed, 22-Mar, 10:15 AM-11:00 AM
Location : Mandalay Bay North, Level 0 – South Pacific A
Presenter(s) : Brian S. Paskin, IBM; David Currie, IBM

HAJ-2718 : Utilizing IBM WebSphere Liberty in Docker Containers for Scalability (Lab)
Date/Time : Wed, 22-Mar, 01:00 PM-02:45 PM
Location : Mandalay Bay South, Level 3 – South Seas H
Presenter(s) : Brian S. Paskin, IBM; David Currie, IBM

BAS-5901 : Choosing a Container Platform for Your WebSphere Applications
Date/Time : Thu, 23-Mar, 10:30 AM-11:15 AM
Location : Mandalay Bay North, Level 0 – South Pacific A
Presenter(s) : David Currie, IBM; Tom Banks, IBM

Meetup Happy

Saturday, July 19th, 2014

I’ve gone a bit meetup happy in the past two weeks. Last week I headed along to the Pivotal offices in London for the first London Cloud Foundry User Group meetup organised by one-time colleague Duncan Winn. First to speaker was another ex-Hursley employee, Glyn Normington. He gave a fascinating presentation in to the work that he and his colleagues are doing to replace the backend of Cloud Foundry’s Warden container with libcontainer (now split out from Docker). More on this over on Glyn’s blog.

Next up was London based Tammer Saleh, Director of Products at Pivotal Cloud Foundry Services. You can see the recording of this session from the Cloud Foundry Summit where they talk about the different models for stacking server instances. Finally, James Watters (Vice President of Product, Marketing and Ecosystem for Cloud Foundry at Pivotal) talked about the roadmap for Cloud Foundry in 2014 (including what’s out of scope). See James Bayer’s session from the summit for similar information.

The next meetup was my first at Agile South Coast. If nothing else, this gave me an excuse to have a nose at the new(ish) Ordnance Survey offices! I can’t claim to have been welcomed with open arms to the group (no-one even commented on the fact that they hadn’t seen me there before) but that’s fine by me. Most notable to me though was the fact that I was the only one there who wasn’t a scrum master by profession. Have developers lost interest in agile?

As one would expect with this audience, it wasn’t long before the post-it notes were out and we were collaborating on choosing subjects to discuss. My heart sunk when topics such as “should spikes be given points?” were selected but I was glad when the resounding response from the group seemed to be “it doesn’t really matter – whatever works for you”. Oh, and apparently PSM is more through than CSM but the latter gets more CV points! As I’m part way through reading Kanban in Action, the discussion on Scrum vs Agile in a BAU environment was interesting. I may yet make it to another of these meet ups.

The American style pizza and good selection of beer certainly helped make the trip into town worthwhile although I’ll not mistakenly pick up the 7.2% Sierra Nevada Torpedo Extra IPA in future!

Lastly, I returned to Developer South Coast for a session entitled “NoSQL vs SQL… Fight!”. Actually, there wasn’t much of a fight to be had as the speaker (Tony Rogerson) is an SQL Server DBA. He gave a thorough although halting coverage of the theory behind relational and NoSQL databases though which sadly meant he ran out of time before reaching the potentially more interesting topic of NewSQL databases.

AWS Update

Monday, May 5th, 2014

At the end of April I went to the AWS Summit at ExCeL London, partly to keep up with the competition but largely because the attendees are a different crowd to those you get at your average IBM conference. I managed to miss most of the keynote, partly by design (no early start and an off-peak ticket) and partly due to someone driving in to a level-crossing in Southampton! Having watched the video subsequently, I don’t think I missed a great deal. The only announcements from Amazon that peaked my interest was the arrival of Amazon WorkSpaces in Ireland and the availability of the Twitter stream in Amazon Kinesis.

As in common with the rest of the day, it was the customer slots that were the most interesting. For example, SwiftKey talked about their use of Hadoop on AWS to crunch Wikipedia in other languages to build a starter set for their language models, through to CloudFront as the CDN for serving the final models up to their customers.

I had an interesting chat over lunch with someone who was actually an IBM customer and then wandered the expo watching demos by some of the likely suspects in the cloud deployment, management and monitoring space (Chef, Splunk, DataDog, …).

After lunch the breakout sessions began with six parallel tracks this year. I went to Deployment Done Right first, covering Elastic Beanstalk, OpsWorks and CloudFormation. The only new news for me was an aside that Beanstalk nows supports Docker. It seems like pretty lame support for containerisation though as you appear to get an EC2 instance per image. The accompanying presentation from Sportpursuit.com was most notable for the long list of open source software in use (Nginx, PHP, Magento, Varnish, Redis, Memcached, Elasticsearch, Jenkins, Capistrano, Capify EC2, Boto, …).

Next up was Dynamic Content Acceleration covering the CloudFront CDN and Route 53 DNS with the aim of knocking a second off your response times. The customer this time was import.io which is an interesting site in its own right, providing the capability to turn websites in to structured data (for free).

For the last session of the day I picked Scaling on AWS for the First 10 Million Users which did not, as you might expect, spend a lot of time on auto-scaling, but covered all aspect of application architecture that would contribute to scaling. The customer was the mobile taxi app firm Hailo who are pursuing a micro-services architecture. They are using containerisation (they didn’t specify which) and are apparently writing their own controller to manage the distribution of those containers across EC2 instances to balance workload.

Logsearch & Decker

Friday, April 4th, 2014

Yesterday evening I headed up to the London PaaS User Group meeting as there were two Cloud Foundry related sessions on the agenda. First up was David Laing talking about the open source Logsearch project, a bosh deploy of an Elastic search ELK log analysis cluster. His employer (City Index) has this hooked up to Cloud Foundry system logs and, in some cases, they’re also using it for analysis of application logs with addition parsers. They’re looking for people to get involved in the project and help with the next phase: anomaly detection. One major hole in the solution as it currently stands: it’s only suitable for private PaaS as their is no access control over the logged data.

Up second was an entertaining pitch by Colin Humphreys, Founder and CEO of ours hosts CloudCredo, on how to sell hats to monkeys. That was the back story anyway, it was actually about how there is space in the stack for something that gives you the flexibility of IaaS over what you run but the simplicity of management, scaling and load balancing of PaaS. That something is Container as a Service. Specifically, the ability to push Docker files to Cloud Foundry using a custom stack for the DEA. Something that Colin is referring to as Decker.

Colin gave a nice demo but it is obviously still early days. Currently you can only push Docker files not images. There is also no staging at the moment – the image is created when each instance starts – consequently it is not taking any advantage of intermediate images. There is obviously lots of scope for improvement and it’s definitely one to watch. It was also interesting that Colin is currently focussing on the Docker side with the DEA interactions set to change with the introduction of Diego. The project is open source but Colin recommended waiting until he writes some docs before you try picking it up!