For reasons that I won’t go in to here, I’ve been interested in the CPU accounting aspect of cgroups for a while and I recently found some time to have a poke at what information is available in the Docker remote API. I was interested in getting hold of the actual CPU time used by a container versus the elapsed time that the container has been running for (where the former would be smaller if the container is not CPU intensive and would potentially be much larger if it’s chewing through multiple cores).
The CLI doesn’t expose the information that I was looking for so my first pass was to define an image with curl and jq:
|
1 2 3 4 |
FROM ubuntu:latest RUN apt-get update \ && apt-get install -y curl jq \ && rm -rf /var/lib/apt/lists/* |
Build it:
|
1 |
docker build -t curl https://git.io/v1ttL |
And then run it with a script as follows:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
#!/bin/bash docker run -v /var/run/docker.sock:/var/run/docker.sock --rm curl bash -c 'names=$(curl -s --unix-socket /var/run/docker.sock http::/containers/json | jq -r ".[] | .Names[0]") \ && for name in $names; do name=${name:1} stats=$(curl -s --unix-socket /var/run/docker.sock http::/containers/$name/stats?stream=false) total_usage=$(echo $stats | jq .cpu_stats.cpu_usage.total_usage) read=$(echo $stats | jq -r .read) read_timestamp=$(date -d $read +%s%3N) started=$(curl -s --unix-socket /var/run/docker.sock http::/containers/$name/json | jq -r .State.StartedAt) started_timestamp=$(date -d $started +%s%3N) elapsed_time=$(($read_timestamp - $started_timestamp)) echo "$name - CPU: $(($total_usage/1000000000))s Elapsed: $(($elapsed_time/1000))s" done' |
I started out with an Alpine based image but the version of date it comes with wasn’t capable of parsing the ISO format dates returned by the API. This was an interesting exercise in the use of curl with Unix sockets and jq for parsing JSON on the command line but I thought I could do better.
Next step was a rendering of the script above in to golang which you can find over on GitHub. You’ll have to forgive my poor golang – I wouldn’t claim to know the language; this is just a cut-and-shut from numerous sources around the internet. Perhaps the only part worth mentioning is that I explicitly pass an empty version string to the golang Docker library so that you don’t get client-server version mismatch errors.
Having compiled this up in to a static binary I could then build a small image from scratch. I then wanted to build this using Docker Hub automated builds and a binary release on GitHub. This raises the thorny issue of how you make the binary executable once you’ve used ADD to download it in to the image. There is one solution here that adds a very small C binary that can be used to perform the chmod. Having initially employed this method it reminded me of another issue that I’d hit. I’d inadvertently doubled the size of our websphere-traditional images to over 3GB with a recursive chmod (the files get copied in to a new layer with the modified permissions). So, in the end I caved in and checked the binary in to GitHub so I could use a COPY and pick up the correct permissions.
The resulting image, weighing it at just over 4MB, is on Docker Hub. As the instructions say, it can be run with the command:
|
1 2 |
docker run -v /var/run/docker.sock:/var/run/docker.sock \ --rm dcurrie/cpu-usage |
To test out the image, let’s spin up a container that should burn up the two cores allocated to my Docker for Mac VM:
|
1 |
docker run -it --rm agileek/cpuset-test |
If we leave it for a few minutes we see an output along the following lines:
|
1 |
drunk_poitras - CPU: 3m34.006087599s Elapsed: 1m47.693831631s |
The total CPU usage is, as we’d expect, twice the elapsed time. Let’s try again but this time run two containers and use cpuset to constrain them both to a single core:
|
1 2 |
docker run -d --cpuset-cpus=0 agileek/cpuset-test docker run -d --cpuset-cpus=0 agileek/cpuset-test |
This time, the results show that each container is getting half of the CPU time:
|
1 2 |
jolly_sammet - CPU: 1m7.47553538s Elapsed: 2m14.894646946s cranky_bardeen - CPU: 1m9.354165997s Elapsed: 2m17.812472231s |
(Actually, you can see that the one that has been running longer has slightly more than half as it got the CPU to itself for a couple of seconds before the other container started!) Finally, and just for interest, let’s spin up an unconstrained WebSphere Liberty server:
|
1 |
docker run -d websphere-liberty:webProfile7 |
After a minute, we see that it’s used just over 20 seconds of CPU time to start up:
|
1 |
goofy_ramanujan - CPU: 21.305670941s Elapsed: 1m3.48256668s |
And if we check again after half an hour, we see that without any load, the server has consumed very little extra CPU:
|
1 |
goofy_ramanujan - CPU: 26.623074485s Elapsed: 30m2.482489205s |