concept health check in category docker

This is an excerpt from Manning's book Learn Docker in a Month of Lunches. Login to get full access to this book.

Run the same test but using the v2 image tag, and leave some time between the commands to let Docker fire the health checks inside the container.

# start the API container, v2
 docker container run -d -p 8081:80 diamol/ch08-numbers-api:v2
 
 # wait 30 seconds or so and list the containers
 docker container ls
 
 # repeat this four times - it returns three random numbers and then fails
 curl http://localhost:8081/rng
 curl http://localhost:8081/rng
 curl http://localhost:8081/rng
 curl http://localhost:8081/rng
 
 # now the app is in a failed state - wait 90 seconds and check
 docker container ls

copy

My output is in figure 8.2. You can see that the new version of the API container initially shows a healthy status--if images have a health check built in, Docker shows the status of the health check for running containers. Some time after I’ve triggered the bug, the container shows as unhealthy.

Figure 8.2 A broken app shows as an unhealthy container, but the container is still up.

That unhealthy status is published as an event from Docker’s API, so the platform running the container is notified and can take action to fix the application. Docker also records the result of the most recent health checks, which you can see when you inspect the container. You’ve already seen the output of docker container inspect , which shows everything Docker knows about the container. If there’s a health check running, that gets shown too.

Pages of JSON data are returned here, and if you scroll to the State field, you’ll see there’s a Health section. That contains the current status of the health check, the “failing streak,” which is the number of successive failures, and a log of the recent health check calls. In figure 8.3 you can see an extract of my container’s state. The health check is in a failing streak of six, which triggers the container to be in an unhealthy state, and you can see the logs from the health check commands, which fail when they get an HTTP status result of 500.

Figure 8.3 Containers with a health check show the health status of the app and the health check logs.

The health check is doing what it should: testing the application inside the container and flagging up to Docker that the app is no longer healthy. But you can also see in figure 8.3 that my unhealthy container has a “running” status, so it’s still up even though Docker knows it is not working correctly. Why hasn’t Docker restarted or replaced that container?

The simple answer is that Docker can’t safely do that, because the Docker Engine is running on a single server. Docker could stop and restart that container, but that would mean downtime for your app while it was being recycled. Or Docker could remove that container and start a new one from the same setup, but maybe your app writes data inside the container, so that would mean downtime and a loss of data. Docker can’t be sure that taking action to fix the unhealthy container won’t make the situation worse, so it broadcasts that the container is unhealthy but leaves it running. The health check continues too, so if the failure is temporary and the next check passes, the container status flips to healthy again.

Health checks become really useful in a cluster with multiple servers running Docker being managed by Docker Swarm or Kubernetes. Then the container platform is notified if the container is unhealthy and it can take action. Because there is additional capacity in a cluster, a replacement container can be started while the unhealthy one is still running, so there shouldn’t be any application downtime.

This is an excerpt from Manning's book Docker in Action, Second Edition. Login to get full access to this book.

8.5.3. The purpose and use of health checks

Health checks are used to determine whether the application running inside the container is ready and able to perform its function. Engineers define application-specific health checks for containers to detect conditions when the application is running, but is stuck or has broken dependencies.

Image authors should define a useful health check in images where possible. Usually this means exercising the application in some way or checking an internal application health status indicator such as a /health endpoint on a web server. However, sometimes it is impractical to define a HEALTHCHECK instruction because not enough is known about how the image will run ahead of time. To address this problem, Docker provides the --health-cmd to define a health check when running a container.

Let’s take the previous HEALTHCHECK example and specify the health check when running the container instead:

docker container run --name=healthcheck_ex -d \
  --health-cmd='nc -vz -w 2 localhost 80 || exit 1' \
  nginx:1.13-alpine

copy

Defining a health check at runtime overrides the health check defined in the image if one exists. This is useful for integrating a third-party image because you can account for requirements specific to your environment.