Archive for the ‘WebSphere’ Category

Using Rocker to build Liberty images from Java source

Saturday, September 24th, 2016

Looking for solutions to my archive extraction problem brought me to look at the Rocker project from Grammarly. I’d looked at it before but not in any great detail. In a nutshell, it aims to extend the Dockerfile syntax to overcome some of its current limitations. Although not an option for me (because I can’t depend anything beyond the standard Docker toolset) the Rocker solution to my earlier problem would be as simple as follows:

The extra syntax here is the MOUNT command which follows the same syntax as the --volume flag on docker run. As the Grammarly team point out, there are trade-offs here which help to explain why the Docker maintainers are reluctant to add volume mounts to docker build. Here, changes to the contents of the mounted directories do not result in the cache being busted.

Anyway, this post is meant to be about a different problem: building Docker images where the chosen language requires compilation e.g. Java. One approach (that taken by OpenShift’s source-to-image) is to add the source to an image that contains all of the necessary pieces to build, package and run it. As shown in Jamie’s WASdev post, for Liberty that might mean putting Maven and a full JDK in to the image. I’m not a fan off this approach: I prefer to end up with an image that only contains what is needed to run the application.

The following shows how this might look using Rocker:

Here we’re building one of the Micro Profile samples (which uses Maven) and then creating an image with the resulting WAR and the new WebSphere Liberty Micro Profile image. You’ll note that there are two FROM statements in the file. First we build on the maven image to create the WAR file. We then use the Rocker EXPORT command to make the WAR file available to subsequent build steps. The second part of the build then starts with the websphere-liberty:microProfile image, imports the WAR and tags it. Building, running and then calling the application is then as simple as follows:

The other thing of note is that we’ve used the MOUNT command to create a volume for the Maven cache. The volume is tied to this Rockerfile so, if you run rocker build --no-cache . you’ll see that it rebuilds the images from scratch but Maven does not need to download the dependencies again.

The MOUNT is also a great way to overcome the long-running issue of how to provide credentials required to access resources at build time without them ending up in the final image. Other nice features include the ATTACH command which effectively allows you to set a breakpoint in your Rockerfile, and the fact that the Rockerfile is pre-processed as a golang template allowing much more advanced substitutions than with Docker’s build arguments.

Unpacking an archive as a non-root user during docker build

Friday, September 23rd, 2016

The way we build our WebSphere traditional Docker images is as a two-step process. First we install the products using Installation Manager and generate a tar file containing the installed product. Then we suck that tar file in to a clean image so that the resulting image does not contain all the cruft left lying around from the install process. (Not including Installation Manager in the final image also helps to reinforce that these images are intended to be immutable.)

The ADD Dockerfile command is very convenient for copying in a tar file from the context and unpacking it, all in one atomic operation. Unfortunately the ADD command ignores the current user and always unpacks as root. (Aside: Docker recently re-opened the proposal created by IBMer Megan Kostick to address this problem.) You could run a chown following the ADD but this results in all the files being copied in to a new layer (not cool when the contents of your tar file weighs in at 1.5GB!). Our initial starting point was to make sure that all the files already had the right ownership when they are added to the tar file. This involved creating the same user/group in the initial image and relying on getting the same uid/guid in the eventual image, something I wasn’t entirely happy with.

A related problem that we had run in to elsewhere was that the copy and unpack magic of ADD doesn’t extend to zip files, a format in which many of our install binaries are found. Where those binaries are already hosted somewhere, it’s simple enough to use wget or curl to pull the files, unpack, and perform any necessary cleanup as part of a RUN command. The obvious solution to my local tar or zip file was to host the file somehow. I decided to spin up a python container to serve up the files as follows:

That URL can then be consumed in a subsequent build. For example, if I had example.tar.gz in the directory on the host, I could unpack as the user/group foo:bar in my image with the following Dockerfile:

To build the image, we then just need to pass in the URL as a build argument and, when we’re done, we can clean up the python container:

The result of all of this is that we then get the default non-root behavior of tar which is to unpack as the current user.

Containerizing background processes

Thursday, September 22nd, 2016

The lifetime of a Docker container is tied to the lifetime of the PID 1 process executed when the container was started. WebSphere Liberty has a convenient server run command to run the application server in the foreground. Sadly, that’s not the case with the traditional WebSphere’s startServer.sh script which simply starts the server process in the background and then exits. To ensure that the container didn’t exit as well, we started out with a script something along the following lines:

where server1.pid is a file created by the server process (but not immediately, hence the initial sleep). That successfully kept the container alive but failed to allow it to shutdown cleanly! A docker stop, for example, would wait for the default timeout period and then kill the process. Not great for any in-flight transactions! The solution was simple enough, add a trap to catch any interrupt and issue the command to stop the server:

All was well with the world until we then enabled server security by default. Unfortunately with security enabled the stopServer.sh script requires credentials to be provided and there is no way to get those credentials to the script. The solution was to switch to sending the interrupt signal to the server process. I also disliked that initial sleep so I decided to retrieve the process ID via ps (something that’s safer in a container given the limited process tree) and then wait whilst the processes directory exists in /proc. The resulting code looked along the following lines:

Note the use of a function so that $PID is not evaluated at the point the trap is set up.
Another disadvantage with having the server process in the background is the lack of output in the container logs. I decided to rectify that whilst I was at it by adding calls to tail the server log files:

The significance of the tail parameters is as follows. The capital F indicates that the attempts to follow the log file should be retried. This ensures that we continue to follow the latest file when the logs roll over. The pid parameter ensures that the background tail processes exit along with the server process. The -n +0 indicates that the output should start at the beginning of the file so that entries output whilst the startServer.sh script is running are not lost. As previously noted, Docker preserves stderr across the remote API so we make sure to direct the output from SystemErr.log there.

WebSphere Liberty and IBM Containers: Part 3

Saturday, June 18th, 2016

Scaling up

In the first two posts in this series we covered the basics of starting a WebSphere Liberty on IBM Containers via the browser and then using the command line to deploy an application.

We’ve already seen some of the value-add that comes out of the box when running under IBM Containers. For example, at no point have we needed to be concerned with the underlying infrastructure on which the containers are running (beyond selecting a region). When we created an image it was scanned automatically for vulnerabilities. Each container was allocated its own private IP address accessible from any other container running in the same space – no need to set up and configure overlay networking here. We had control over whether we also wanted to assign a public IP and, if so, what ports should be exposed there. We also had easy access to metrics and standard out/error from the container.

So far we’ve only deployed a single container though. What happens when we hit the big time and need to scale up our application to meet demand? When we created our first container via the UI, you may remember that the Single option was selected at the top. Let’s go back and try out the Scalable alternative. From the catalog, navigate through Compute and Containers (remember that these instructions are based on the new Bluemix console). Select our existing demo image. Next, select the Scalable button at the top and give the container group a name. By default you’ll see that our group will contain two instances.

Rather than having a single IP associated with a container, this time we are asked to give a host name (on the mybluemix.net domain by default) for the group. Requests arriving at this host name will be load-balanced across the container instances in the group (reusing the gorouter from Cloud Foundry). One nice bonus of this approach is that it doesn’t eat in to our quota of public IPs! As the host name needs to be unique within the domain, I tend to include my initials as you’ll see in the screenshot below. Select 9080 as the HTTP port and then click Create.

Container group creation

Once the containers have started, the dashboard should show that you have two instances running:

Running instances

Right-click on the route shown in the Group details section and open it in a new tab. This should take you to a Liberty welcome page and, if you add the myLibertyApp context root, you should be able to see the application again. If you hit refresh a few times, although you won’t be able to tell with this application, your requests will be load-balanced across the two instances. If you return to the dashboard and switch to the Monitoring and Logs tab you can switch between the output for the instances and should, for example, be able to see the spike in network usage on the two containers when you made the requests.

If you return to the Overview tab you will see that there are plus and minus symbols either side of the current number of instances. These can be used to manually scale the number of instances. Click the + icon, click Save, and watch the creation of a new container in the group.

Manual scaling is all very well but it would be better if the number of instances scaled automatically up and down as required. If you’re deploying your containers in the London region then you’ll notice an extra tab at the top of the dashboard labelled Auto-Scaling. It’s only available in the London region at the moment because the service is still in beta (and so things may change a bit from what I’m describing here). Having selected this tab, click the plus icon labelled Create policy. Give the policy a name default and set the minimum and maximum instance values to 1 and 3. Add two CPU usage rules to scale up and down the number of instances as shown in the following diagram and then hit Create. Finally, select Attach to activate the policy for this scaling group.

Auto-Scaling

If you click the Auto-Scaling History tab you should see that a scaling action has taken place. We originally scaled up manually to 3 instances but, as the CPU usage is below our 60% limit, the number gets scaled down by one. If you wait another 5 minutes (the cool down period we specified), then you’ll see it get scaled down again to our minimum of 1.

Scaling history

And that concludes our tour of the scaling options in IBM Containers!

WebSphere Liberty on IBM Containers: Part 2

Monday, May 30th, 2016

Deploying an Application

In the first part of this series we looked at how to get started running a WebSphere Liberty image in IBM Containers using the Bluemix console.  The container was just running an empty Liberty server. In this post we’ll look at building and deploying an image that adds an application. I was originally intending to stick to using the browser for this post but I think it’s actually easier to do this from the command line. I’m going to assume that you already have Docker installed locally, either natively on Linux, via Docker Machine, or via the Docker for Mac/Windows beta.

First off we need an application to deploy and, just for novelty, I’m going to use the Liberty app accelerator to generate one. Select Servlet as the technology type and then, contrary as it may seem, select Deploy to Local and not Deploy to Bluemix. The latter option currently only supports deploying to the Instant Runtimes (Cloud Foundry) side of Bluemix. Finally, give your project a name and click Download Now.

Liberty App Accelerator

Unpack the zip file you downloaded and change to the top directory of the project. The app is built using maven. Perhaps you already have maven installed but this is a Docker blog post so we’re going to use the maven image from Docker Hub to build the app as follows:

$ docker run –rm -v $(pwd):/usr/src/mymaven \
    -w /usr/src/mymaven/myProject-application maven mvn clean package

This mounts the project on to a container running the maven image and runs the command mvn clean package in the myProject-application directory. Note: if you were doing this repeatedly you’d probably want to mount a maven cache in to the image as well and not download everything each time)

In the myProject-application/target directory you should now find that you have a file myArtifactId-application-1.0-SNAPSHOT.war. Copy this in to a new empty directory so that when we execute a Docker build we don’t end up uploading lots of other cruft to the Docker engine. Using your favourite editor, add the following Dockerfile to the same directory:

FROM websphere-liberty:webProfile7
COPY myArtifactId-application-1.0-SNAPSHOT.war /config/dropins

We have two choices now, we can either build a Docker image locally and then push that up to the IBM Containers registry, or we can build the image in IBM Containers. We’ll go for the latter option here as it involves pushing less bytes over the network.

There’s one niggle today that, to access your IBM Containers registry, you need to log in first using the Cloud Foundry CLI and IBM Containers plugin. We’re going to play the containerisation trick again here. Run the following commands to build an image with the CLI and plugin:

$ docker build -t cf https://git.io/vr7pl

Ideally I’d run this image as a stateless container but getting the right state written out to host in to the .cf, .ice and .docker directories is a bit finicky. Instead, we’re going to mount our current directory on to an instance of the image and perform the build inside:

$ docker run -it –rm $(pwd):/root/build cf
$ cd /root/build
$ cf login -a api.ng.bluemix.net

$ cf ic login
$ cf ic build -t demo .

Now we’re ready to run an instance of your newly built image. At this point you could switch back the to the UI but lets keep going with the command line. We’ll need to refer to the built image using the full repository name, including your namespace:

$ ns=$(cf ic namespace get)
$ cf ic run –name demo -P registry.ng.bluemix.net/$ns/demo

By default, containers are only assigned a private IP address. In order to access our new container we’ll need to request and assign a public IP. The cf ic ip command unfortunately returns a human friendly message, not a computer friendly one, hence the need for the grep/sed to retrieve the actual IP:

$ ip=$(cf ic ip request | grep -o ‘”.*”‘ | sed ‘s/”//g’)
$ cf ic ip bind $ip demo

Lastly, we can list the port and IP to point our browser at:

$ cf ic port demo 9080

Adding the root context myLibertyApp should give use the welcome page for the starter app.

Starter Welcome Page

Congratulations, you’ve successfully deployed an application to IBM Containers! In the next post in this series we’ll look at some of the additional features that the service provides, such as scaling groups and logging.

WebSphere Liberty on IBM Containers: Part 1

Monday, May 16th, 2016

Getting Started

I wanted write a few blog posts as I explored deploying our WebSphere Liberty Docker image on to third-party cloud platforms but this didn’t really feel fair given that I haven’t written anything here about the options using IBM’s cloud platform! So, to kick things off, here’s a series of posts on using Liberty with the IBM Containers service in Bluemix. In this post I’m going to focus on using the web UI to get a simple Liberty container up and running. In subsequent posts I’ll cover topics such as deploying an application, scaling applications and using the command line.

I’m also going to use the new console for the instructions/screenshots in this post. If you don’t already have an account, use the sign-up button at the top right of the console to get a free 30-day trial.

You can use WebSphere in a number of different ways in Bluemix: web applications deployed as Cloud Foundry applications will use the Liberty runtime and it’s also possible to stand-up WAS ND topologies for both Liberty and the traditional application server in VMs. Here we’re going to focus on containers.

Once logged in, select the Compute icon and then the Containers tab. Then click the + icon top-right to create a new container. You should find that the private registry associated with your account is already populated with a number of IBM-provided images as shown in the following screen shot. Select the ibmliberty image. This image adds a few layers on top of the websphere-liberty image from Docker Hub to make it run more cleanly in IBM Containers (see the Dockerfile in the docs for full details).

Select container image

You will now see a drop-down on the left which allows you to select the image tag to use. By default the latest tag will give you the full Java EE7 profile image. Select the webProfile7 tag to get a lighter-weight image. Beneath the tag you’ll an area entitled Vulnerability Assessment which, if we’ve been doing our properly, should say Safe to Deploy. Assessments are updated regularly as new CVEs are announced and we therefore regularly refresh the image to make sure that it stays clean.

Vulnerabilty Advisor

At the top of the page, you’ll see that the Single option is selected to create just a single container instance, we’ll return later to see how the Scalable option works. Give the container a name. Leave the container size at Micro but select the option to Request and Bind Public IP.  All containers get an IP address on a private network and are accessible by other containers in the same space but to be accessible externally they must be assigned a public IP.

Create a container

Now click the Create button at the bottom of the page. After a minute or two the status of the container should change to Running.

Container running

Switch to the Monitoring and Logs tab. The container level metrics around CPU/memory/network usually take some time to appear but if you select the Logging tab you should be able to see the messages output by Liberty as the container started. Tip: you can use the ADVANCED VIEW button on these tabs to open up Grafana and Kibana views on to these metrics and logs respectively if you want to do more detailed analysis.

Container logs

Finally, switch back to the Overview tab. If you scroll down to the Container details section, you’ll find a list of ports. If you click on 9080, this will open a new browser tab using that port and the public IP address assigned to the container. With the default image, that should show the Liberty welcome page.

Liberty welcome page

Congratulations – you have run WebSphere Liberty on IBM Containers! The astute among you will, however, have noticed that it’s not running any application. That’ll be the subject of the next blog post…

WebSphere Appliance Management Center

Tuesday, April 24th, 2012

I haven’t posted anything product related since the release of WebSphere ESB 7.5 in June last year where I hinted at pastures new. Those of you that I’m connected to on LinkedIn will know that I subsequently became Technical Lead for a product called WebSphere Appliance Management Center. WebSphere Appliance Management Center provides off-box management and monitoring of multiple WebSphere DataPower SOA appliances. The development for the management component of the product moved to Hursley shortly after the initial release (named 4.0 after the coincident firmware release). The new team put together a fix pack later in the year which delivered support for the new Service Gateway XG45 appliance.

Today saw the announcement of the next chapter in the WebSphere Appliance Management Center story. Significantly, you will find this in the WebSphere DataPower Appliances firmware V5.0 announcement letter as the next version will no longer be a chargeable product in its own right. Instead, it will be freely downloadable and supported free of charge for all customers with a current support entitlement for a WebSphere DataPower SOA appliance.

Not only will it cost you less – but we’ll also be giving you less! The team has been working flat out to create a much lighterweight offering which is faster to install and less resource intensive. The management user interface has also been extensively reworked to be more responsive and better support user interaction patterns. In particular, the current restrictions around managed set membership will be lifted allowing much great flexibility for firmware and configuration deployment. In addition to the existing domain management capabilities, you will also be able to manage configuration at the service level. All of this will be available for download on June 26.

For those attending IBM Impact next week who’d like to find out more about this exciting new release, I’ll be co-presenting on WebSphere Appliance Management Center (Monday, 5:15-6:30pm, Lando 4305). We will also be hosting a series of round table sessions in Toscana 3701 (Monday, 10:45-12:00; Tuesday, 1:20-2:45pm; Thursday 3:15-4:30pm) which will be your chance to shape the future direction of the product. Alternatively, please feel free to contact me directly or post questions in the DataPower developerWorks forum.

WebSphere ESB 7.5 available

Friday, June 3rd, 2011

WebSphere ESB 7.5 is generally available from Passport Advantage as of today (as are WSRR 7.5, WebSphere ESB Registry Edition 7.5 and the 7.5 BPM stack) and the InfoCenter is also live. Having been the Development Lead and then Release Architect, I feel a certain sense of parental pride in this delivery but, as always, the credit goes to the wider team. It shall also be my last as I am parting company with the WebSphere ESB development team and moving on to pastures new. I can’t say more at the moment but all will be revealed sometime this month.