Graceful shutdown stops an application from accepting new work while allowing work already in progress to finish. Kubernetes uses this approach when replacing Pods, upgrading nodes, or stopping workloads, reducing failed requests and abandoned jobs. section 10.1.2 section 12.1.3
Implement graceful shutdown as a coordinated application-and-Kubernetes process:
- Handle SIGTERM in application code and begin the shutdown sequence. section 10.1.2 section 12.1.3
- Stop accepting new requests or jobs while completing the current request or task. section 10.1.2 section 12.1.3
- Configure terminationGracePeriodSeconds with enough time for normal completion; long-running jobs may need a much longer window. section 10.1.2 section 12.1.3
- Use readiness checks separately to prevent traffic from reaching Pods before they are ready; readiness controls startup, while graceful termination controls stopping. section 12.1.3
Worker Deployment
A Python worker registers a SIGTERM handler that changes its running state, uses a five-second queue-pop timeout to notice that state, finishes its current calculation, and exits. Its Deployment sets terminationGracePeriodSeconds: 120, giving the worker two minutes to complete in-progress work. section 10.1.2
Listing 10.6 Chapter10/pi_worker2/pi_worker.py
import os
import signal
import redis
from pi import *
redis_host = os.environ.get('REDIS_HOST')
assert redis_host != None
r = redis.Redis(host=redis_host,
port='6379',
decode_responses=True)
running = True #1
def signal_handler(signum, frame): #2
print("got signal") #2
running = False #2
signal.signal(signal.SIGTERM, signal_handler) #2
print("starting")
while running:
task = r.blpop('queue:task', 5) #3
if task != None:
iterations = int(task[1])
print("got task: " + str(iterations))
pi = leibniz_pi(iterations)
print (pi)
#1 Instead of looping indefinitely, exit the loop if this variable is set to false by the signal handler.
#2 Register a signal handler to set the running state to false when SIGTERM is received.
#3 Pops the next task but now only waits for 5 seconds if the queue is empty (to allow the running condition to be checked)
Inspect the worker implementation to see how a signal handler changes the loop condition and lets the current task finish before process exit. section 10.1.2
