A health check lets a task report whether it is operating normally, so an orchestrator can react before a single failed instance makes an API unavailable. The task owns the definition of healthy; the manager only calls the declared endpoint and interprets the result. section 9.0 section 9.4.3
In a simple task-level design, each task configuration supplies a stable health-check path such as /health. The manager combines that path with the worker address and the Docker-assigned host port, issues an HTTP request, and treats 200 as healthy. A connection error or any other status marks the check as unsuccessful. section 9.4 section 9.4.3
Health checking has two complementary responsibilities:
- The worker inspects each container through Docker’s ContainerInspect API and changes a task to Failed when its container is no longer running. section 9.4
- The manager periodically checks running tasks and sends a restart event when a health check fails; tasks already marked Failed are also restarted. section 9.4 section 9.4.3
- Restart bookkeeping is explicit: RestartCount increases for every restart, and checks or restarts stop after three attempts. section 9.4.3
Listing 9.7 The manager’s new checkTaskHealth method
func (m *Manager) checkTaskHealth(t task.Task) error {
log.Printf("Calling health check for task %s: %s\n",
t.ID, t.HealthCheck)
w := m.TaskWorkerMap[t.ID] #1
hostPort := getHostPort(t.HostPorts) #2
worker := strings.Split(w, ":") #3
url := fmt.Sprintf("http://%s:%s%s",
worker[0], *hostPort, t.HealthCheck) #4
log.Printf("Calling health check for task %s: %s\n", t.ID, url)
resp, err := http.Get(url) #5
if err != nil {
msg := fmt.Sprintf("Error connecting to health check %s", url)
log.Println(msg)
return errors.New(msg) #6
}
if resp.StatusCode != http.StatusOK { #7
msg := fmt.Sprintf("Error health check for task %s did not
return 200\n", t.ID)
log.Println(msg)
return errors.New(msg)
}
log.Printf("Task %s health check response: %v\n", t.ID, resp.StatusCode)
return nil
}
#1 Gets the worker where the task is running from the manager’s TaskWorkerMap
#2 Retrieves the port on which the task is listening
#3 Extracts the IP address from the string representation of the worker stored in the TaskWorkerMap
#4 Builds a URL to the health check using the worker’s IP address, the port on which the task is listening, and the health check endpoint defined in the task definition
#5 Calls the health check
#6 Handles connection errors
#7 Handles non-200 responses
The checkTaskHealth implementation makes the contract concrete: locate the scheduled worker, obtain the mapped host port, build the task URL, perform GET, and return an error unless the response is 200. section 9.4.3
A periodic loop performs these checks every 60 seconds. Restarting first resets the task state to Scheduled, increments its restart count, stores the updated task, and posts a task event to the worker. If the worker cannot be reached, the task is placed in a pending queue instead. section 9.4.3
Echo-service success and failure
An echo service exposes /health, which returns 200, and /healthfail, which returns 500. A task configured with /health remains healthy when the manager reaches its mapped port. A task configured with /healthfail fails the check and is restarted until RestartCount reaches 3. Even a normally healthy endpoint triggers a restart when the manager can no longer connect after a container’s port changes. section 9.5 section 9.5
This approach deliberately separates task health from worker health. A successful task endpoint is enough for the manager’s task-restart decision, while determining whether the worker component itself is functioning remains a separate operational concern. section 9.4
