A termination grace period is the time Kubernetes gives a Pod to shut down cleanly after receiving SIGTERM. It lets an application finish current work instead of being stopped abruptly. section 10.1.2 section 12.1.3
A reliable shutdown combines two pieces:
- Handle SIGTERM in the application, stop accepting new work, and finish the current request or task. section 10.1.2 section 12.1.3
- Set terminationGracePeriodSeconds long enough for that work to complete; Kubernetes uses the configured window before forced termination. section 10.1.2 section 12.1.3
- Choose the duration for the workload: short-lived web requests need time to drain, while long-running jobs may need a much longer window. section 12.1.3
Worker Pod with a two-minute window
A background worker receives SIGTERM, stops taking new queue jobs, and finishes its current task. Setting terminationGracePeriodSeconds: 120 gives it up to 120 seconds to exit cleanly during a Deployment update or node upgrade. section 10.1.2
Listing 10.6 Chapter10/pi_worker2/pi_worker.py
import os
import signal
import redis
from pi import *
redis_host = os.environ.get('REDIS_HOST')
assert redis_host != None
r = redis.Redis(host=redis_host,
port='6379',
decode_responses=True)
running = True #1
def signal_handler(signum, frame): #2
print("got signal") #2
running = False #2
signal.signal(signal.SIGTERM, signal_handler) #2
print("starting")
while running:
task = r.blpop('queue:task', 5) #3
if task != None:
iterations = int(task[1])
print("got task: " + str(iterations))
pi = leibniz_pi(iterations)
print (pi)
#1 Instead of looping indefinitely, exit the loop if this variable is set to false by the signal handler.
#2 Register a signal handler to set the running state to false when SIGTERM is received.
#3 Pops the next task but now only waits for 5 seconds if the queue is empty (to allow the running condition to be checked)
The worker listing shows a SIGTERM handler that changes the loop state so the process can finish its current iteration and exit. section 10.1.2
