liveBook · Manning

Executable runbook

As explained in Build High Value Software Systems

An operational procedure whose diagnostic steps are implemented as code that can run automatically and present results for human decisions.

An executable runbook turns operational steps into code: it runs diagnostics, presents results, and leaves decisions to people where judgment is needed. Unlike prose that can describe an outdated process, code that breaks after a service or tool changes demands attention. section 12.4

Listing 12.4 Example executable runbook for payment processing outage
def payment_outage_runbook():
    """Executable runbook: payment processing outage."""
    print("Step 1: Checking payment gateway status...")
    gw_status = check_gateway_health("stripe")
    print(f"  Gateway status: {gw_status}")
 
    print("Step 2: Checking payment-service error rate...")
    error_rate = get_service_error_rate("payment-service", window="5m")
    print(f"  Error rate: {error_rate:.2%}")
 
    print("Step 3: Checking database connection pool...")
    pool_usage = get_db_pool_usage("payments-db")
    print(f"  Pool usage: {pool_usage:.0%}")
 
    if pool_usage > 0.90:
        print("  ACTION: Pool near capacity. Kill idle connections.")
        kill_idle_connections("payments-db", older_than_minutes=10)
 
    print("Step 4: Checking queue depth...")
    depth = get_queue_depth("payment-processing-queue")
    print(f"  Queue depth: {depth} (normal < 100)")
 
    if depth > 500:
        print("  ESCALATE: Queue backing up. Page payments team lead.")
        page_oncall("payments-team-lead", "Queue depth critical: {depth}")

A payment outage runbook checks gateway health, payment-service errors, database connection-pool use, and queue depth, with an action for a pool near capacity. section 12.4

Keep the procedure useful as operations change:

  • Have the service-owning team convert a frequently used runbook, then assess it during an incident. section 12.4
  • Treat missing steps or decision points revealed by incidents as updates to the runbook code. section 12.4
  • Use runbooks during incidents; use pipeline constraints to check conditions before changes ship. section 12.4

How Build High Value Software Systems covers it

How deep it goes
Explains automated steps, human decision points, team ownership, and incident-driven maintenance.
Examples
Uses a payment outage to check gateway health, service errors, database pool use, and queue depth.
What you can do after
Build it from scratch by converting a frequently used procedure, then updating its code when incidents expose missing steps.
How it connects
Links runbooks to incident response, kaizen, standard work, and deployment pipelines.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build High Value Software Systems. The section links point to the parts of the book it draws on.