David Currie

For a while now I’ve had a bullet point on a chart that blithely stated that you could add an application on top of our WebSphere Application Server traditional Docker image using wsadmin and, in particular, with the connection type set to NONE i.e. without the server running. Needless to say, when I actually tried to do this shortly before a customer demo it didn’t work! Thankfully, with moral support from a colleague and the excellent command assistance in the admin console, it turns out that my initial assertion was correct and the failure was just down to my rusty scripting skills. Here’s how…

First, the Dockerfile that builds an image containing our ferret sample application, taking the WAR file from Maven Central.

The following script then builds an image from the Dockerfile in the gist above, runs it, waits for the server to start, and then retrieves the ferret webpage.

Looking for solutions to my archive extraction problem brought me to look at the Rocker project from Grammarly. I’d looked at it before but not in any great detail. In a nutshell, it aims to extend the Dockerfile syntax to overcome some of its current limitations. Although not an option for me (because I can’t depend anything beyond the standard Docker toolset) the Rocker solution to my earlier problem would be as simple as follows:

The extra syntax here is the MOUNT command which follows the same syntax as the --volume flag on docker run. As the Grammarly team point out, there are trade-offs here which help to explain why the Docker maintainers are reluctant to add volume mounts to docker build. Here, changes to the contents of the mounted directories do not result in the cache being busted.

Anyway, this post is meant to be about a different problem: building Docker images where the chosen language requires compilation e.g. Java. One approach (that taken by OpenShift’s source-to-image) is to add the source to an image that contains all of the necessary pieces to build, package and run it. As shown in Jamie’s WASdev post, for Liberty that might mean putting Maven and a full JDK in to the image. I’m not a fan off this approach: I prefer to end up with an image that only contains what is needed to run the application.

The following shows how this might look using Rocker:

Here we’re building one of the Micro Profile samples (which uses Maven) and then creating an image with the resulting WAR and the new WebSphere Liberty Micro Profile image. You’ll note that there are two FROM statements in the file. First we build on the maven image to create the WAR file. We then use the Rocker EXPORT command to make the WAR file available to subsequent build steps. The second part of the build then starts with the websphere-liberty:microProfile image, imports the WAR and tags it. Building, running and then calling the application is then as simple as follows:

The other thing of note is that we’ve used the MOUNT command to create a volume for the Maven cache. The volume is tied to this Rockerfile so, if you run rocker build --no-cache . you’ll see that it rebuilds the images from scratch but Maven does not need to download the dependencies again.

The MOUNT is also a great way to overcome the long-running issue of how to provide credentials required to access resources at build time without them ending up in the final image. Other nice features include the ATTACH command which effectively allows you to set a breakpoint in your Rockerfile, and the fact that the Rockerfile is pre-processed as a golang template allowing much more advanced substitutions than with Docker’s build arguments.

The way we build our WebSphere traditional Docker images is as a two-step process. First we install the products using Installation Manager and generate a tar file containing the installed product. Then we suck that tar file in to a clean image so that the resulting image does not contain all the cruft left lying around from the install process. (Not including Installation Manager in the final image also helps to reinforce that these images are intended to be immutable.)

The ADD Dockerfile command is very convenient for copying in a tar file from the context and unpacking it, all in one atomic operation. Unfortunately the ADD command ignores the current user and always unpacks as root. (Aside: Docker recently re-opened the proposal created by IBMer Megan Kostick to address this problem.) You could run a chown following the ADD but this results in all the files being copied in to a new layer (not cool when the contents of your tar file weighs in at 1.5GB!). Our initial starting point was to make sure that all the files already had the right ownership when they are added to the tar file. This involved creating the same user/group in the initial image and relying on getting the same uid/guid in the eventual image, something I wasn’t entirely happy with.

A related problem that we had run in to elsewhere was that the copy and unpack magic of ADD doesn’t extend to zip files, a format in which many of our install binaries are found. Where those binaries are already hosted somewhere, it’s simple enough to use wget or curl to pull the files, unpack, and perform any necessary cleanup as part of a RUN command. The obvious solution to my local tar or zip file was to host the file somehow. I decided to spin up a python container to serve up the files as follows:

That URL can then be consumed in a subsequent build. For example, if I had example.tar.gz in the directory on the host, I could unpack as the user/group foo:bar in my image with the following Dockerfile:

To build the image, we then just need to pass in the URL as a build argument and, when we’re done, we can clean up the python container:

The result of all of this is that we then get the default non-root behavior of tar which is to unpack as the current user.

The lifetime of a Docker container is tied to the lifetime of the PID 1 process executed when the container was started. WebSphere Liberty has a convenient server run command to run the application server in the foreground. Sadly, that’s not the case with the traditional WebSphere’s startServer.sh script which simply starts the server process in the background and then exits. To ensure that the container didn’t exit as well, we started out with a script something along the following lines:

where server1.pid is a file created by the server process (but not immediately, hence the initial sleep). That successfully kept the container alive but failed to allow it to shutdown cleanly! A docker stop, for example, would wait for the default timeout period and then kill the process. Not great for any in-flight transactions! The solution was simple enough, add a trap to catch any interrupt and issue the command to stop the server:

All was well with the world until we then enabled server security by default. Unfortunately with security enabled the stopServer.sh script requires credentials to be provided and there is no way to get those credentials to the script. The solution was to switch to sending the interrupt signal to the server process. I also disliked that initial sleep so I decided to retrieve the process ID via ps (something that’s safer in a container given the limited process tree) and then wait whilst the processes directory exists in /proc. The resulting code looked along the following lines:

Note the use of a function so that $PID is not evaluated at the point the trap is set up.
Another disadvantage with having the server process in the background is the lack of output in the container logs. I decided to rectify that whilst I was at it by adding calls to tail the server log files:

The significance of the tail parameters is as follows. The capital F indicates that the attempts to follow the log file should be retried. This ensures that we continue to follow the latest file when the logs roll over. The pid parameter ensures that the background tail processes exit along with the server process. The -n +0 indicates that the output should start at the beginning of the file so that entries output whilst the startServer.sh script is running are not lost. As previously noted, Docker preserves stderr across the remote API so we make sure to direct the output from SystemErr.log there.