Chapter 5: Testing as waste prevention
The chapter introduction shows how tests can either prevent rework or create waste, depending on the risks they protect and the costs they impose.
After this chapter you will be able to
- Risk-aligned testing value. Explain why tests should be judged by whether they protect important behavior and prevent rework, rather than by how many tests or how much coverage a suite has.
- Value and risk of tests. Explain why a test earns its keep by protecting customer, business, or developer value and reducing real risk, rather than merely adding coverage. Describe the section’s three checks: whether it catches a production-relevant defect, fails only when behavior breaks, and runs often enough to be useful.
- Testing waste and flow signals. Describe how misaligned tests create waste beyond defects, and explain why observing CI and developer behavior reveals test-suite health better than coverage percentages alone.
- Behavior versus implementation tests. Distinguish a test tied to an implementation detail from one that protects customer-visible behavior, using the payment-processor rate-limiter refactor. Explain why the cost of that distinction grows across a large suite.
- Upstream Defect Prevention. Explain why introducing a test constraint before implementation is cheaper than detecting a defect in CI or production, and contrast test-first prevention with test-after detection.
- Behavioral Tests Shape Design. Describe how defining expected behavior before implementation shapes the design, including when AI generates the code.
- Useful Unit Test Controls. Distinguish unit tests that protect behavior from tests that police implementation details, and explain the effects on feedback and refactoring.
- Risk and detection cost. Explain why testing decisions should consider a defect’s probability and impact as well as how long it may go undetected, rather than coverage alone.
- Test layers as containment zones. Describe which risks unit, integration, and end-to-end tests should contain and why faster layers should catch suitable failures before they reach slower layers or production.
- Risk matched testing investment. Justify concentrating deterministic tests on likely or high-impact failures while avoiding excessive tests where they slow feedback or add little protection.
- Unreliable feedback and normalized failures. Explain why a flaky test is a Lean defect and how repeated reruns can make red builds lose their meaning.
- Trust in red builds. Explain how repeated false alarms make red builds less meaningful and change how developers respond to failures.
- Waste from unreliable assertions. Explain how one flaky assertion creates several kinds of waste beyond rerun time.
- Flakiness and larger batches. Trace why unreliable test feedback leads developers to push larger batches and how that changes cycle time and failure diagnosis.
- Activity ratio as system signal. Explain what the activity ratio reveals in the example and how a team could use it to target flow improvements rather than judge a developer.
- Structural coverage versus business value. Explain why AI-generated tests can cover many code paths without protecting the behavior that matters to users or the business.
- Behavioral contracts over internal wiring. Distinguish a test of the gold-member discount rule from tests that pin down its implementation, and describe the cost when safe refactors break the latter.
- Risk filter for generated tests. Apply the section’s value filter to a proposed AI-generated test and explain how a clear risk model changes AI’s contribution.
- Early defect feedback. Explain why the section advocates automating feedback and give an example of how automation shortens a feedback loop.
- Value aligned automation. Explain why the decision to automate a test depends on whether it prevents more waste than it creates, rather than whether automation is possible.
- CI pipeline value stream. Explain how a CI pipeline turns a code change into a deployable, verified artifact, and why each stage should be considered in terms of risk reduction or delay.
- Risk aligned automation. Distinguish automation that protects known risks at the right granularity and layer from automation that accumulates without intention and becomes a bottleneck.
- Coverage versus business protection. Explain that coverage records which code ran, not whether tests would catch a consequential defect. Use the payment-service example to show how an assertion about a declined payment can protect a contract while barely changing the coverage percentage.
- Risk guided coverage use. Describe how to use coverage to find untested paths in high-impact modules, then judge which gaps matter through the value lens rather than pursuing a target percentage.
- Commit flow and activity ratio. Trace a real commit from first edit to merge, estimate its cycle time and active work time, and calculate the activity ratio to see whether waiting dominated.
- Testing delays and protection. Identify the longest delay in a commit’s journey and distinguish test failures or automated checks that protect meaningful value from those that add instability or noise.
- Single change and remeasurement. Apply one concrete change to a source of testing waste, then measure the next commit for effects on cycle time, the meaning of red builds, or refactoring safety.
What it builds on
- Value-adding versus non-value-adding work (chapter 2) · Applies the distinction between value-adding and non-value-adding work to tests.
- Types of value in software (chapter 2) · The distinction among user, business, and technical value.
- Defects (chapter 2) · Defects as a form of software waste.
- The eight wastes in software development (chapter 2) · The Lean categories of waste.
- Gemba and Kaizen in software development (chapter 2) · Gemba as observation of work where it happens.
- Design for deletion (chapter 4) · The aim of making code easier to change or delete.
- Spec first: Define before you build (chapter 4) · The practice of defining intent before building.
- Designing for intervention: Enabling the software Andon (chapter 4) · The idea of an Andon signal that prompts intervention.
Section by section
Introduction
The chapter introduction shows how tests can either prevent rework or create waste, depending on the risks they protect and the costs they impose.
- Risk-aligned testing value: Explain why tests should be judged by whether they protect important behavior and prevent rework, rather than by how many tests or how much coverage a suite has.
Why it matters: This gives you a way to judge whether a test earns the effort it takes to create and maintain. - Brittle tests and refactoring waste: Describe how tests that assert internal structure rather than business outcomes can fail during a behavior-preserving change and force needless updates.
Why it matters: A suite can look thorough while making safe changes slower rather than safer. - Unreliable and delayed feedback: Explain how flaky tests and slow CI each make test feedback less useful to a team.
Why it matters: Feedback helps prevent rework only when the team can trust it and act on it promptly.
Key terms: risk, rework, value, confidence, internal structure, business outcomes, mocks, overproduced test, flaky test, red builds
5.1 Value-add versus wasteful testing
This section distinguishes tests that protect valuable behavior from tests that create waste, using test-suite flow and a payment-processor refactor to show the difference.
- Value and risk of tests: Explain why a test earns its keep by protecting customer, business, or developer value and reducing real risk, rather than merely adding coverage. Describe the section’s three checks: whether it catches a production-relevant defect, fails only when behavior breaks, and runs often enough to be useful.
Why it matters: These checks help you decide whether a test prevents meaningful rework or adds to it. - Testing waste and flow signals: Describe how misaligned tests create waste beyond defects, and explain why observing CI and developer behavior reveals test-suite health better than coverage percentages alone.
Why it matters: Looking at how the suite affects work exposes costs that a coverage percentage can hide. - Behavior versus implementation tests: Distinguish a test tied to an implementation detail from one that protects customer-visible behavior, using the payment-processor rate-limiter refactor. Explain why the cost of that distinction grows across a large suite.
Why it matters: The example shows how tests can obstruct a safe change instead of protecting the outcome that matters.
Key terms: value, real risk, defect, behavior, every commit, overproduction, waiting, motion, overprocessing, inventory
5.2 Tests as design constraints
This section explains how tests written before implementation constrain design and prevent defects more cheaply than tests added afterward.
- Upstream Defect Prevention: Explain why introducing a test constraint before implementation is cheaper than detecting a defect in CI or production, and contrast test-first prevention with test-after detection.
Why it matters: Where you place a test affects both the cost of correction and the length of the feedback loop. - Behavioral Tests Shape Design: Describe how defining expected behavior before implementation shapes the design, including when AI generates the code.
Why it matters: Stating the desired behavior first gives both developers and AI-generated code a boundary to work within. - Useful Unit Test Controls: Distinguish unit tests that protect behavior from tests that police implementation details, and explain the effects on feedback and refactoring.
Why it matters: Behavior-focused tests can provide fast, trustworthy feedback without making refactoring unnecessarily difficult.
Key terms: upstream prevention, downstream detection, building quality in, test-first, design constraint, behavior, AI, unit tests, implementation policing, contracts
5.3 Structuring test layers around risk
This section explains how to place tests in layers according to the failures they can contain, the value at risk, and the cost of detecting defects late.
- Risk and detection cost: Explain why testing decisions should consider a defect’s probability and impact as well as how long it may go undetected, rather than coverage alone.
Why it matters: It helps you spend testing effort where early detection protects the most value. - Test layers as containment zones: Describe which risks unit, integration, and end-to-end tests should contain and why faster layers should catch suitable failures before they reach slower layers or production.
Why it matters: Assigning each risk to an appropriate layer catches defects sooner without making every test a full-system test. - Risk matched testing investment: Justify concentrating deterministic tests on likely or high-impact failures while avoiding excessive tests where they slow feedback or add little protection.
Why it matters: Matching effort to risk avoids both escaped defects and test suites that obstruct feedback.
Key terms: risk, impact, probability, detection time, unit tests, integration tests, end-to-end tests, containment zones, testing pyramid, high-probability logic failures
5.4 Identifying flaky tests as flow-blocking defects
Defines flaky tests and explains how their unreliable results turn test failures into ambiguous signals that disrupt development flow.
- Flaky test behavior: Describe what makes a test flaky and give an example of a result changing even though the code under test has not changed.
Why it matters: Recognizing flakiness helps distinguish an unreliable test result from a change in the code under test. - Unreliable feedback and normalized failures: Explain why a flaky test is a Lean defect and how repeated reruns can make red builds lose their meaning.
Why it matters: When teams stop trusting red builds, testing no longer provides dependable feedback for their work.
Key terms: flaky test, nondeterministic test, feedback loop, red builds, defect
5.4.1 When signals lose meaning
This section shows how a flaky test can erode trust in CI failures, change developer behavior, and create waste beyond the cost of reruns.
- Trust in red builds: Explain how repeated false alarms make red builds less meaningful and change how developers respond to failures.
Why it matters: A failure signal only helps protect quality when people trust it enough to act on it. - Time coupled expiration test: Describe why the order-expiration test can pass locally yet fail intermittently in CI.
Why it matters: The example shows how an unreliable assertion can produce a failure that a rerun appears to clear. - Waste from unreliable assertions: Explain how one flaky assertion creates several kinds of waste beyond rerun time.
Why it matters: The cost of a flaky test includes wasted effort and missed defects, not just a slower build.
Key terms: red builds, flaky tests, Andon cord, expiration worker, pending, flaky, Waiting, Motion, Overprocessing, Inventory
5.4.2 Flow metrics
This section uses flow metrics to show how flaky tests and waiting extend delivery time, encourage batching, and reveal opportunities to improve the system.
- Three measures of delivery flow: Distinguish lead time, cycle time, and activity time by their start and end points and by whether waiting counts.
Why it matters: Separating these measures shows whether a delay comes from doing the work or waiting for it to move. - Flakiness and larger batches: Trace why unreliable test feedback leads developers to push larger batches and how that changes cycle time and failure diagnosis.
Why it matters: A flaky test can slow far more than one test run by changing how a team delivers and diagnoses changes. - Activity ratio as system signal: Explain what the activity ratio reveals in the example and how a team could use it to target flow improvements rather than judge a developer.
Why it matters: The ratio helps a team find where work waits instead of simply asking people to work faster.
Key terms: lead time, cycle time, activity time, touch time, flakiness, single-piece flow, batching, variability, activity ratio, Gemba
5.5 Inflating coverage without protecting value
This section shows why high test coverage can still create waste, and how to direct AI-generated tests toward business contracts rather than implementation details.
- Structural coverage versus business value: Explain why AI-generated tests can cover many code paths without protecting the behavior that matters to users or the business.
Why it matters: A large suite is not necessarily a safer suite. - Behavioral contracts over internal wiring: Distinguish a test of the gold-member discount rule from tests that pin down its implementation, and describe the cost when safe refactors break the latter.
Why it matters: Tests should alert you to a broken promise, not make safe changes expensive. - Risk filter for generated tests: Apply the section’s value filter to a proposed AI-generated test and explain how a clear risk model changes AI’s contribution.
Why it matters: The selection filter determines whether cheap test generation reduces risk or adds upkeep.
Key terms: structural exploration, path coverage, business value, overproduction, business contract, pin the wiring, pricing rule, flow damage, value filter, risk model
5.6 Deciding what not to automate
Automation is valuable when it shortens feedback loops and catches defects before they travel further.
- Early defect feedback: Explain why the section advocates automating feedback and give an example of how automation shortens a feedback loop.
Why it matters: Fast feedback helps catch problems before they move further through delivery.
Key terms: automation, feedback loops, defects
5.6.1 Waste elimination and creation
This section shows how to decide whether a test is worth automating by comparing the waste it prevents with the waste it creates.
- Value aligned automation: Explain why the decision to automate a test depends on whether it prevents more waste than it creates, rather than whether automation is possible.
Why it matters: The test suite should protect valuable outcomes without becoming a source of delay and noise. - Checkout contract protection: Justify automating the checkout validations on every commit using the business risks, test cost, and refactoring example.
Why it matters: The checkout example makes the waste-prevention decision concrete. - Behavior over implementation assertions: Distinguish the valuable checkout behavior tests from the additional assertion the team decides not to automate.
Why it matters: Choosing behavior-focused checks avoids maintenance work that does not protect customer outcomes.
Key terms: automation, waste, flow, inventory, meaningful contracts, checkout service, payment gateway, deterministic, business logic, overautomation
5.6.2 Automation as a value stream
This section treats CI automation as a value stream whose stages should protect meaningful risks without creating avoidable delays.
- CI pipeline value stream: Explain how a CI pipeline turns a code change into a deployable, verified artifact, and why each stage should be considered in terms of risk reduction or delay.
Why it matters: Viewing CI as a value stream makes its delays as visible as its checks. - Risk aligned automation: Distinguish automation that protects known risks at the right granularity and layer from automation that accumulates without intention and becomes a bottleneck.
Why it matters: Automation helps flow when developers can act on its signals without waiting for unnecessary checks.
Key terms: CI pipeline, value stream, deployable, verified artifact, granularity, layer, known risk, meaningful failures, bottleneck
5.7 Coverage metrics: Expose risk, not inflate confidence
This section shows why high code coverage can coexist with unprotected business behavior and how to use coverage to find meaningful testing gaps.
- Coverage versus business protection: Explain that coverage records which code ran, not whether tests would catch a consequential defect. Use the payment-service example to show how an assertion about a declined payment can protect a contract while barely changing the coverage percentage.
Why it matters: A green coverage report cannot tell you whether the behavior customers rely on is protected. - Risk guided coverage use: Describe how to use coverage to find untested paths in high-impact modules, then judge which gaps matter through the value lens rather than pursuing a target percentage.
Why it matters: Using coverage as a signal rather than a target directs testing effort toward failures that matter.
Key terms: code coverage, business contract, test protected intent, coverage, high-impact modules, value lens, blind spots
5.8 Exercise: Expose and reduce one source of testing waste
This exercise uses a real commit to make testing waste visible, remove one source of it, and measure the effect.
- Commit flow and activity ratio: Trace a real commit from first edit to merge, estimate its cycle time and active work time, and calculate the activity ratio to see whether waiting dominated.
Why it matters: Separating active work from elapsed time makes delays visible. - Testing delays and protection: Identify the longest delay in a commit’s journey and distinguish test failures or automated checks that protect meaningful value from those that add instability or noise.
Why it matters: The longest delay and the checks that cause it are better improvement targets than testing in general. - Single change and remeasurement: Apply one concrete change to a source of testing waste, then measure the next commit for effects on cycle time, the meaning of red builds, or refactoring safety.
Why it matters: Changing one thing and measuring afterward shows whether the intervention actually helped.
Key terms: cycle time, activity time, activity ratio, waiting, flaky tests, meaningful value, automated checks, concrete intervention, testing waste, red builds
5.9 Summary
Where this is used later
- Reducing batch sizes for faster feedback (chapter 6) · Develops the relationship between batch size and faster feedback.
- Lean metrics for developers (chapter 9) · The chapter develops these flow measures further.
- The metrics value stream (chapter 6) · The metrics value stream examines elapsed time and active progress in delivery.
- Measurement for learning versus ranking (chapter 9) · Measurement for learning relies on treating metrics as system signals rather than individual rankings.
- CI/CD: Flow enabler or disruptor? (chapter 6) · Supports evaluating whether CI/CD helps or disrupts flow.
- Attacking a legacy codebase systematically (chapter 7) · Distinguishing a useful coverage signal from a claim that coverage proves quality.
- Cycle time (chapter 9) · Cycle-time measurement builds on tracing the elapsed time of a change.
- What Kaizen actually means (chapter 10) · This small, repeated improvement cycle supports the later discussion of Kaizen.
Concepts explained in this chapter
- code coverageA measure of which code elements, such as lines, branches, or functions, are executed during testing.
- software testingA procedure that evaluates software behavior against expected results under specified conditions.
- unit testAn automated test that verifies the behavior of a small isolated unit of software.
- test layerA level in a testing strategy that checks software for a particular class of failures at a characteristic speed and scope.
- activity ratioThe proportion of elapsed work time spent on productive activity rather than waiting or other non-progressing periods.
- wasteWork that creates no customer value and can be eliminated without harming delivery or its outcomes.
- CPU timeThe time spent actively progressing a change through activities such as coding, testing, fixing, and reviewing.
- test suiteA collection of tests executed to evaluate software behavior or quality.
- single-piece flowA production approach in which one item moves through the system at a time so defects can be detected and corrected before the next item is produced.
- integration testA test that verifies interactions among software components or between software and an external system.
- riskThe possibility of harm or loss, assessed here through the probability and impact of a failure and the time it remains undetected.
- overproduction wasteLean waste caused by producing more, earlier, or sooner than demand requires.
- flaky testA test whose result is nondeterministic despite unchanged software and test inputs.
- test-driven developmentA development practice in which failing tests are written before implementation and then used to guide code until the tests pass.
- automationThe use of software or machines to perform tasks with reduced manual intervention.
- valueThe benefit users receive from software that meets their needs through usable, reliable behavior.
- cycle timeThe elapsed time required for work to move through a production or delivery process.
- batch sizeThe number of work items grouped together for processing or delivery as one batch.
- jidokaA Lean production approach that builds quality into processes by stopping when abnormalities occur and preventing defects from proceeding.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
