liveBook · Manning

Chapter 12: Lean operations and production

What you will learn in chapter 12 of Build High Value Software Systems

A deployment scenario shows why Lean operations must look beyond successful delivery and green dashboards to how the system actually runs.

After this chapter you will be able to

What it builds on

Section by section

Introduction

A deployment scenario shows why Lean operations must look beyond successful delivery and green dashboards to how the system actually runs.

Key terms: Lean operations, p99 latency, error rate

12.1 Shifting from development to operations mindset

This section shifts the measure of a finished feature from passing tests and deploying successfully to serving users reliably in production over time.

Key terms: development, operations, defective inventory, behavior under load, unit tests, database connection pool, green CI pipeline, production, gemba, value stream

12.1.1 Development velocity versus operational stability

This section argues that development speed must be balanced with production stability and shows how Lean thinking shifts from the development workflow to the running system.

Key terms: development velocity, operational stability, overproduction, flow with quality, change failure rate, mean time to recovery, availability, latency distribution, p95, p99

12.2 Gemba in production

This section shows how to observe a running production system directly to find problems that dashboards and alerts can miss.

Key terms: gemba, running system, dashboard, p50 latency, p99, production logs, retry storms, connection pool churn, single request, end-to-end

12.2.1 The silent degradation pattern

The section explains how production changes can silently degrade user experience, why observing production reveals failures dashboards miss, and how mistake-proofing can prevent their recurrence.

Key terms: silent degradation, error rates, gemba, dashboards, request thread, worker thread, poka-yoke, mistake proofing, structural vulnerabilities, CI check

12.3 Andon for production systems

The section explains how production alerts can act as andon signals by pointing responders toward causes, providing diagnostic context, and matching notification urgency to impact.

Key terms: operational alerting, resource, consumer, shared dependency, symptom, causal context, runbook, actionable, tier alerts by urgency, page

12.3.1 The cost of not stopping

This section explains why production andon must turn alerts into prompt action, why hesitation is costly, and how teams can use internal reliability expectations to make delays visible.

Key terms: andon, signal, stop, MTTR, canary rollout, 99.99% availability, four nines, error budget, Service Level Agreement (SLA), Service Level Expectation (SLE)

12.3.2 Pulling the cord

Pulling the andon cord in production means using a cause-aware signal to choose a prepared, targeted response rather than simply stopping a system that is serving customers.

Key terms: andon cord, rollback, circuit breaker, cause-aware alerts, analytics batch job, currency exchange API, one-click action, andon cords, failure mode, degrees of running

12.3.3 Three levels of andon

This section distinguishes three levels of production andon by who detects, stops, and recovers from a problem, and explains when automated recovery is safe.

Key terms: andon, signal, stop, recover, blast radius, Level 2, Level 3, confidence in the diagnosis, auto-remediating, automatic rollback

12.4 Standard work for operations and incident response

This section shows how executable runbooks and automated pipeline constraints turn operational experience into standard work that improves after incidents and prevents repeat mistakes.

Key terms: production waste, standard work, runbooks, incident response procedures, pipeline constraints, executable runbook, diagnostic steps, human judgment, service-owning team, incident remediation

12.5 Mura, Muri, Muda

This section explains how uneven production demand creates both overload during peaks and wasted capacity during lulls, and why those risks can be easy to miss.

Key terms: mura, muri, muda, overburden, time series data, steady state metrics, headroom

12.5.1 Mura: The unevenness that causes outages

This section explains how uneven demand causes production incidents and how workload leveling can prevent peaks from overwhelming a system without reducing total work.

Key terms: mura, peak-to-trough ratio, muri, muda, heijunka, leveling, decouple acceptance from processing, batch workloads, rate limiter, geographic load distribution

12.5.2 Muri: The overburden that breaks things

This section explains how production systems become overburdened when fluctuating demand meets insufficient capacity, and why headroom and steadier demand reduce outage risk.

Key terms: muri, overburden, capacity, cascading timeouts, headroom, 60–80% utilization, critical services, mura, leveled demand

12.5.3 Muda: Capturing operational waste

This section shows how to detect waste in production and feed its signals back into development so teams can eliminate the waste at its source.

Key terms: TIMWOODS, operational waste, feedback loops, development value stream, waiting, overprocessing, queue, API response, rolled-back deployment, defect

12.6 Production Kaizen

This section shows how production teams use repeated, measured Kaizen cycles to improve operational outcomes by addressing the current bottleneck with an appropriate Lean tool.

Key terms: production kaizen, gemba, andon, standard work, mura, muri, MTTR, diagnosis, remediation, andon Level 3

12.6.1 Post-incident reviews as kaizen triggers

This section shows how post-incident reviews turn production failures into concrete, tracked system improvements.

Key terms: kaizen trigger, post-incident review, poka-yoke, executable runbook, andon, muda, standard work, production kaizen, kaizen card, baseline measurement

12.7 Exercise: Audit your production system through the Lean operations lens

This exercise guides a production audit that turns direct observations and operational evidence into one measured kaizen cycle.

Key terms: gemba, production system, andon, cord, executable runbook, standard work, mura, muri, peak-to-trough ratio, operational waste

12.8 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 11: Architectures that flow Chapter 13: Lean team culture →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.