An executable runbook turns operational steps into code: it runs diagnostics, presents results, and leaves decisions to people where judgment is needed. Unlike prose that can describe an outdated process, code that breaks after a service or tool changes demands attention. section 12.4
Listing 12.4 Example executable runbook for payment processing outage
def payment_outage_runbook():
"""Executable runbook: payment processing outage."""
print("Step 1: Checking payment gateway status...")
gw_status = check_gateway_health("stripe")
print(f" Gateway status: {gw_status}")
print("Step 2: Checking payment-service error rate...")
error_rate = get_service_error_rate("payment-service", window="5m")
print(f" Error rate: {error_rate:.2%}")
print("Step 3: Checking database connection pool...")
pool_usage = get_db_pool_usage("payments-db")
print(f" Pool usage: {pool_usage:.0%}")
if pool_usage > 0.90:
print(" ACTION: Pool near capacity. Kill idle connections.")
kill_idle_connections("payments-db", older_than_minutes=10)
print("Step 4: Checking queue depth...")
depth = get_queue_depth("payment-processing-queue")
print(f" Queue depth: {depth} (normal < 100)")
if depth > 500:
print(" ESCALATE: Queue backing up. Page payments team lead.")
page_oncall("payments-team-lead", "Queue depth critical: {depth}")A payment outage runbook checks gateway health, payment-service errors, database connection-pool use, and queue depth, with an action for a pool near capacity. section 12.4
Keep the procedure useful as operations change:
- Have the service-owning team convert a frequently used runbook, then assess it during an incident. section 12.4
- Treat missing steps or decision points revealed by incidents as updates to the runbook code. section 12.4
- Use runbooks during incidents; use pipeline constraints to check conditions before changes ship. section 12.4
