Chapter 8: Root cause thinking for developers
The introduction uses a recurring payment-service failure to show why developers should trace evidence to a changeable root cause instead of repeatedly restoring service with temporary fixes.
After this chapter you will be able to
- Symptom fixes versus root causes. Distinguish a temporary restoration of service from a root-cause fix, using the payment-service failure to explain why the errors kept returning.
- Evidence-led cause tracing. Explain how root cause thinking approaches a failure: trace cause chains, check what an initial explanation misses, and follow evidence to something that can be changed permanently.
- Root Cause Investigation Decision. Explain when a root cause investigation is justified and state the two questions that guide it.
- Three Layer Root Cause Resolution. Describe the fix, prevent, and detect layers using the payment-service integer-column example, and explain why the migration alone is insufficient.
- Practical scope of Lean tools. Explain why the section focuses on four Lean problem-solving tools rather than the full toolbox, and why they are suited to software incidents.
- Five Whys reasoning chain. Trace how each answer to a why question becomes the basis for the next question, stopping at something changeable. Explain why the upload example calls for a size limit at the API boundary rather than a bigger queue.
- Evidence behind each answer. Explain why each answer in a 5 Whys chain must be checked against observed behavior, and what to do when an answer cannot be verified.
- Domain-specific cause categories. Describe how a fishbone places a problem at the head and branches potential causes into categories, then explain why software teams should choose categories suited to the problem rather than treat the manufacturing 6Ms as fixed.
- Structured category sweep. Explain how checking every cause category can challenge an initial hunch, using the staging deployment failure to show how the investigation shifted from Process to Technology.
- Frequency-based problem prioritization. Explain how to use Pareto analysis to choose which recurring software problem to tackle first, and illustrate the payoff with the database-timeout example.
- Prioritization versus root cause. Distinguish what Pareto analysis identifies from what a root cause investigation must still establish.
- Failure boundary comparison. Explain how Is/Is Not analysis contrasts where a failure occurs with where it does not, using the differences to narrow down its cause.
- Combined root cause analysis. Explain why combining Is/Is Not, 5 Whys, fishbone diagrams, and Pareto analysis produces a more complete root cause analysis than using any one tool or merely writing a post-incident document.
- Notification failure cause chain. Trace how the notification example moves from the observed failures to the migration-spec cause, rather than treating a service restart as an explanation.
- Observation before theory. Explain what the software gemba is and why debugging should begin with observed system behavior rather than a theory based on code or a stack trace.
- Evidence led RCA tools. Describe how the observed failure determines which root-cause analysis tool to use first rather than following a fixed tool sequence.
- Request flow versus crash frame. Distinguish the exception location shown by a stack trace from the request flow and system state needed to find a root cause. Explain this distinction using the preferences-row example.
- System recovery and detection. Describe how to verify a deployed countermeasure at the gemba and what detection to add afterward.
- Retry Versus Root Cause. Trace why the intermittent database test fails and distinguish the AI-suggested retry from the root cause fix.
- Human Diagnosis Before AI Fixes. Explain why AI patch suggestions need human root cause analysis and how to assess a suggestion before applying it.
- Durable countermeasure criteria. Distinguish a countermeasure from a quick fix using the fix-prevent-detect pattern: fixing the immediate defect is not enough unless the change also prevents recurrence or detects the condition early.
- Safer default ways of working. Explain why strong countermeasures often update standard work, automation, or other system constraints as well as code, so future work is safer by default.
- One-page A3 discipline. Explain why an A3 keeps the problem, cause, and fix on one page, and why its medium and section layout can vary.
- Problem framing and evidence. Describe how the opening parts of an A3 establish what happened, its measurable impact, a measurable goal, and evidence for the root cause.
- Countermeasures and follow-through. Describe how an A3 connects specific countermeasures to accountable implementation, verification, and broader learning.
- Root causes beyond code. Explain why root cause thinking applies to product and operational problems as well as code problems.
- Issue selection and A3 analysis. Apply the exercise to a recent production issue or, if none is available, a recurring backlog bug. Describe how to write a one-page A3 using the eight-section structure while retaining Root Cause Analysis and Countermeasures and combining RCA tools as needed.
- Countermeasure testing and follow-through. Describe how to test each proposed countermeasure, handle quick fixes, and follow through after writing the A3.
What it builds on
- Prioritizing waste by impact (chapter 7) · Prioritizing problems by their impact.
- Establishing your AI standard work (chapter 3) · Uses the idea of standard work as an explicit guide for recurring development work.
- Technical debt (chapter 7) · Relates unaddressed temporary fixes to technical debt in existing code.
Section by section
Introduction
The introduction uses a recurring payment-service failure to show why developers should trace evidence to a changeable root cause instead of repeatedly restoring service with temporary fixes.
- Symptom fixes versus root causes: Distinguish a temporary restoration of service from a root-cause fix, using the payment-service failure to explain why the errors kept returning.
Why it matters: Restoring service can hide a fault that will trigger the same failure again. - Evidence-led cause tracing: Explain how root cause thinking approaches a failure: trace cause chains, check what an initial explanation misses, and follow evidence to something that can be changed permanently.
Why it matters: A plausible explanation is not enough if it leads only to another workaround.
Key terms: symptom-fix cycle, underlying fault, root cause, root cause thinking, structured questioning techniques, cause chains
8.1 Going beyond symptoms to the root cause
This section explains when to investigate a recurring failure and how to address its root cause rather than repeatedly fixing its symptoms.
- Root Cause Investigation Decision: Explain when a root cause investigation is justified and state the two questions that guide it.
Why it matters: It helps you decide when repeated fixes warrant a deeper investigation. - Three Layer Root Cause Resolution: Describe the fix, prevent, and detect layers using the payment-service integer-column example, and explain why the migration alone is insufficient.
Why it matters: Fixing the present crash is not enough if the same mistake—or an unanticipated boundary case—can still reach production.
Key terms: root cause thinking, symptom fix, fix, prevent, detect, schema migration, CI lint, observability check
8.2 Lean problem-solving tools and techniques
This introduction explains why a small set of practical Lean tools is enough for most software root-cause investigations.
- Practical scope of Lean tools: Explain why the section focuses on four Lean problem-solving tools rather than the full toolbox, and why they are suited to software incidents.
Why it matters: You can approach common incidents with a manageable set of tools without first mastering the entire Lean toolbox.
Key terms: Lean problem-solving toolbox, gemba, root cause investigations
8.2.1 The 5 whys
This section shows how to use the 5 Whys to move from an incident symptom to a changeable cause, while checking every answer against evidence.
- Five Whys reasoning chain: Trace how each answer to a why question becomes the basis for the next question, stopping at something changeable. Explain why the upload example calls for a size limit at the API boundary rather than a bigger queue.
Why it matters: Following the chain can reveal a change that addresses the cause instead of merely easing a symptom. - Evidence behind each answer: Explain why each answer in a 5 Whys chain must be checked against observed behavior, and what to do when an answer cannot be verified.
Why it matters: An unsupported answer can make every later conclusion in the investigation unreliable.
Key terms: 5 Whys, countermeasure, API boundary, missing requirement, observed behaviour, logs, metrics, traces, reproducible test, evidence
8.2.2 Fishbone diagrams
This section explains how a fishbone diagram broadens a software investigation across possible causes before the team narrows its focus.
- Domain-specific cause categories: Describe how a fishbone places a problem at the head and branches potential causes into categories, then explain why software teams should choose categories suited to the problem rather than treat the manufacturing 6Ms as fixed.
Why it matters: Useful categories help you consider causes outside the area you first suspect. - Structured category sweep: Explain how checking every cause category can challenge an initial hunch, using the staging deployment failure to show how the investigation shifted from Process to Technology.
Why it matters: Checking the categories you might skip can prevent hours spent pursuing the wrong suspected cause.
Key terms: fishbone, Ishikawa diagram, cause and effect diagram, spines, 6Ms, structured sweep, Process, Technology, runtime
8.2.3 Pareto analysis
This section explains how Pareto analysis ranks recurring software failures so teams investigate the highest-impact problem first.
- Frequency-based problem prioritization: Explain how to use Pareto analysis to choose which recurring software problem to tackle first, and illustrate the payoff with the database-timeout example.
Why it matters: Focusing on the largest category can remove more failures than spreading effort across smaller ones. - Prioritization versus root cause: Distinguish what Pareto analysis identifies from what a root cause investigation must still establish.
Why it matters: A frequent symptom is a useful starting point, but its cause still needs investigation.
Key terms: Pareto analysis, rank by frequency, top contributor, 5 Whys, fishbone, root cause
8.2.4 Is/Is not
This section explains how Is/Is Not analysis uses the boundaries of an inconsistent failure to narrow down its cause.
- Failure boundary comparison: Explain how Is/Is Not analysis contrasts where a failure occurs with where it does not, using the differences to narrow down its cause.
Why it matters: Clear boundaries turn a seemingly random failure into a more focused investigation. - Dimension selection for elusive failures: Describe which dimensions to compare when a failure cannot be reproduced locally, and explain how the checkout example narrows the investigation.
Why it matters: Choosing useful comparisons can reveal the condition missing from a local reproduction.
Key terms: Is/Is Not analysis, boundaries, dimensions, environment, timing, user segment, platform, data characteristics, feature flag
8.2.5 Combining them: Root cause analysis
The section shows how four Lean tools work together to narrow a problem, trace its cause, check for related failures, and prioritize a fix.
- Combined root cause analysis: Explain why combining Is/Is Not, 5 Whys, fishbone diagrams, and Pareto analysis produces a more complete root cause analysis than using any one tool or merely writing a post-incident document.
Why it matters: Combining the tools helps you avoid stopping at a plausible explanation while missing related problems or a higher-impact fix. - Notification failure cause chain: Trace how the notification example moves from the observed failures to the migration-spec cause, rather than treating a service restart as an explanation.
Why it matters: A restart may restore delivery temporarily without explaining why the queue failed. - Related failures and impact: Describe how the fishbone check broadened the investigation beyond the initial cause chain and how Pareto analysis identified the impact of fixing the mappings.
Why it matters: Checking for related conditions and their frequency can turn a fix for one visible error into a higher-impact change.
Key terms: Root Cause Analysis (RCA), Is/Is Not, 5 Whys, fishbone, Pareto, Pro plan, EU region, company_logo, migration spec, Process
8.3 Debugging at the gemba
This section shows how to debug at the software gemba by observing the running system, finding the conditions behind a failure, and verifying that a root-cause fix works.
- Observation before theory: Explain what the software gemba is and why debugging should begin with observed system behavior rather than a theory based on code or a stack trace.
Why it matters: Observing first keeps a plausible but incomplete theory from driving the fix. - Production conditions in reproduction: Explain why a production-only bug must be investigated under the conditions that produce it, or in a staging environment close enough to those conditions.
Why it matters: Reproduction is useful only if it preserves the conditions that make the bug appear. - Evidence led RCA tools: Describe how the observed failure determines which root-cause analysis tool to use first rather than following a fixed tool sequence.
Why it matters: Choosing from the evidence makes RCA tools useful for the failure at hand. - Request flow versus crash frame: Distinguish the exception location shown by a stack trace from the request flow and system state needed to find a root cause. Explain this distinction using the preferences-row example.
Why it matters: Following the flow can uncover the cause of bad state rather than only handling its visible exception. - System recovery and detection: Describe how to verify a deployed countermeasure at the gemba and what detection to add afterward.
Why it matters: Watching the system recover tests whether the countermeasure worked and helps detect a return of the condition.
Key terms: software gemba, observability tooling, reproduce at the gemba, production environment, staging setup, Is/Is Not, 5 Whys, fishbone, Pareto, stack trace
8.4 When AI introduces non-RCA fixes
This section shows how an AI-generated patch can hide a bug’s symptom and argues for human root cause analysis before using AI to produce a fix.
- Retry Versus Root Cause: Trace why the intermittent database test fails and distinguish the AI-suggested retry from the root cause fix.
Why it matters: A passing test can still leave the underlying defect in place for every test using the same helper. - Human Diagnosis Before AI Fixes: Explain why AI patch suggestions need human root cause analysis and how to assess a suggestion before applying it.
Why it matters: This keeps AI’s speed from turning a visible failure into a harder-to-detect recurring defect.
Key terms: retry loop, race condition, create_test_user(), await, 5 Whys, RCA, gemba, hypothesis, fix-prevent-detect test, symptom fix
8.5 Countermeasures, not quick fixes
The section explains how to turn root cause analysis into durable countermeasures rather than leaving a quick fix in place.
- Durable countermeasure criteria: Distinguish a countermeasure from a quick fix using the fix-prevent-detect pattern: fixing the immediate defect is not enough unless the change also prevents recurrence or detects the condition early.
Why it matters: Finding a cause matters only if the resulting change reduces the chance or impact of another failure. - Safer default ways of working: Explain why strong countermeasures often update standard work, automation, or other system constraints as well as code, so future work is safer by default.
Why it matters: Changing the default way work happens protects future changes, not just the instance that failed. - Temporary fix follow-through: Describe when a quick fix is justified and how to keep it from becoming permanent debt; explain how old workaround comments can identify unfinished root cause work.
Why it matters: Tracking a temporary repair keeps an urgent response from quietly becoming long-term operational debt. - Countermeasures at external boundaries: Explain how to respond when the root cause belongs to a provider, partner team, or library you do not control.
Why it matters: An external cause does not leave your team powerless to reduce its effects.
Key terms: countermeasure, quick fix, fix-prevent-detect, observability and alerting, standard work, automated checks, process changes, architectural changes, quick fixes, countermeasures
8.6 A3 reports: Structured, blameless problem solving
This section explains how a concise, blameless A3 report communicates a root cause analysis, its countermeasures, and the plan for checking whether they worked.
- One-page A3 discipline: Explain why an A3 keeps the problem, cause, and fix on one page, and why its medium and section layout can vary.
Why it matters: A concise A3 lets others grasp and evaluate the fix without a meeting or a long document. - Problem framing and evidence: Describe how the opening parts of an A3 establish what happened, its measurable impact, a measurable goal, and evidence for the root cause.
Why it matters: A reader can assess a proposed fix only when the impact, target, and reasoning behind the diagnosis are visible. - Countermeasures and follow-through: Describe how an A3 connects specific countermeasures to accountable implementation, verification, and broader learning.
Why it matters: A diagnosis becomes useful when the team can act on it and check whether the changes last. - Blameless system-focused framing: Explain why an A3 describes process or system gaps instead of blaming the person associated with an incident.
Why it matters: People are more likely to surface failures when the report seeks system improvements rather than someone to fault. - Consequential incidents and institutional memory: Distinguish incidents that warrant an A3 from routine fixes, and describe what a collection of A3s can reveal over time.
Why it matters: Selective reporting preserves the effort for consequential problems while building a record the team can learn from.
Key terms: A3 report, one page, Root Cause Analysis, Countermeasures, Background, Current state, Goal, Implementation plan, Follow-up, Learnings
8.7 RCA beyond your value stream
The section extends root cause thinking beyond code to product decisions and operational processes, showing why fixing a persistent cause matters more than polishing a symptom.
- Root causes beyond code: Explain why root cause thinking applies to product and operational problems as well as code problems.
Why it matters: It helps you look for the source of a recurring problem rather than repeatedly treating its visible effects. - Product naming behind low adoption: Trace how the low-adoption example shifts attention from onboarding to a product naming decision.
Why it matters: The example shows how an apparent UX problem can originate in a product decision. - Database coupling behind deployment delays: Explain how the deployment example distinguishes scheduling trouble from the dependency that causes it.
Why it matters: Comparing affected and unaffected deployments reveals what actually creates the delay.
Key terms: root cause thinking, symptoms, causes, 5 Whys, onboarding flow, naming, product decision, Is/Is Not analysis, shared database, coupling
8.8 Exercise: Write an A3 report for a production issue
The exercise guides readers to write a concise A3 report for a real issue, test its countermeasures, and share and act on the analysis.
- Issue selection and A3 analysis: Apply the exercise to a recent production issue or, if none is available, a recurring backlog bug. Describe how to write a one-page A3 using the eight-section structure while retaining Root Cause Analysis and Countermeasures and combining RCA tools as needed.
Why it matters: A concrete issue and structured analysis make the report useful for addressing an actual problem. - One-page analytical clarity: Explain why the A3 must fit on one page and what to do if it does not.
Why it matters: Concise writing forces the reasoning to become clearer without discarding important findings. - Countermeasure testing and follow-through: Describe how to test each proposed countermeasure, handle quick fixes, and follow through after writing the A3.
Why it matters: The report only helps prevent recurrence when proposed fixes are scrutinized, made visible, and acted on.
Key terms: A3 report, Root Cause Analysis, Countermeasures, RCA tools, one-page A3, sharp enough, countermeasure, quick fix, root cause, A3
8.9 Summary
Where this is used later
- Tracking decisions (chapter 11) · Looking beyond an incident's symptoms to its root cause.
- Muda: Capturing operational waste (chapter 12) · Looking beyond symptoms for an underlying cause.
- A3 as a communication API for improvements (chapter 10) · A3 is revisited as a way to communicate improvements.
Concepts explained in this chapter
- cause analysisA systematic process for identifying the underlying causes of a problem or failure rather than addressing only its symptoms.
- fix-prevent-detect testA completeness pattern for addressing a problem by fixing its immediate effects, preventing recurrence, and detecting conditions that still arise.
- countermeasureA countermeasure is an action designed to address a problem’s cause and prevent or reduce its recurrence.
- is/is not analysisA problem-analysis technique that compares where, when, and under what conditions a problem occurs with conditions where it does not.
- Five WhysA root-cause analysis technique that repeatedly asks why a problem occurred, using each answer to guide the next question.
- fishbone diagramA cause-analysis diagram that places a problem at the head of a central spine and branches potential causes into categories.
- A3 reportA structured Lean problem-solving approach that documents an issue, its analysis, and proposed countermeasures in a concise report.
- pareto analysisA prioritization technique that ranks causes by frequency or impact so that attention can focus on the few causes responsible for most problems.
- standard workA single prescribed method applied across teams regardless of differences in their work contexts.
- AI-generated codeSource code produced by an artificial-intelligence system from supplied inputs or instructions.
- system boundaryA boundary is a separation between system parts, representations, ownership areas, or organizational responsibilities across which information or work passes.
- flaky testA test whose result is nondeterministic despite unchanged software and test inputs.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
