Chapter 9: Measuring what matters
The chapter introduction frames metrics as a way to expose hidden system problems and judge whether measured improvement is real.
After this chapter you will be able to
- Visibility into system problems. Explain why teams measure software delivery: to make otherwise invisible stalls, queues, and system-created drag visible in data they can act on.
- Real improvement versus metric gaming. Distinguish a useful metric from a harmful one by asking whether improving its number actually improves the system, rather than rewarding busywork or dashboard gaming.
- System Versus Station Measurement. Explain why a team's coding speed or deployment frequency can give a misleading picture of delivery. Describe why the customer-facing time from request to production is the more useful measure here.
- Hidden Time in Delivery. Describe what the fintech team's measurements revealed about its 23-day lead time and why the section treats the gap between lead time and activity time as a system problem.
- System patterns behind metrics. Explain why a developer's metric may reflect the delivery system rather than individual performance, using the on-call cycle-time example.
- Fear and problem signals. Explain how attaching reported problems or metrics to individual performance judgments affects whether developers make system problems visible.
- Management by means versus results. Distinguish using numerical targets to judge people from using shared metrics to understand and improve the processes producing results.
- Diagnostic signals versus targets. Explain why using a metric as a target can make the number improve while reliable delivery gets worse, using the build-time example.
- Process-based metric improvement. Describe how a team should respond to an unfavorable metric and why a numerical target needs a method for changing the process.
- Developer view of flow metrics. Explain that lead time, cycle time, activity time, and activity ratio measure how value moves through a system, and that developers can apply this flow perspective to tickets, PRs, and the deploy pipeline.
- End-to-end lead time. Explain what lead time measures and why fast coding alone may not mean fast delivery.
- Typical and extreme lead times. Distinguish what the median and p90 reveal about completed tickets, and explain why an average can misrepresent typical flow when there is a large outlier.
- Control chart variation signals. Explain how a lead-time control chart distinguishes ordinary variation from work items worth investigating.
- Outlier investigation over targets. Describe the section's starting approach to measuring lead time, investigating slow outliers, and avoiding lead-time targets.
- Cycle time boundaries. Explain what cycle time measures, how it differs from lead time, and why a team must agree on when work starts and counts as complete.
- Prework queue time gap. Explain what the gap between lead time and cycle time reveals and what kind of improvement a large gap suggests.
- Control charts and phase diagnosis. Describe how comparing lead-time and cycle-time charts, then decomposing unusually long cycle times, helps locate a constraint.
- Learning rather than developer ranking. Justify using cycle-time charts to investigate work items rather than using individual cycle-time targets to rank developers.
- Active work versus waiting. Distinguish activity time from elapsed time for a work item, giving examples of active progress and passive waiting.
- Hidden queues within stations. Explain how comparing activity time with elapsed time at a process station exposes waiting that a value stream map may not show.
- Activity ratios and improvement experiments. Describe how to use activity ratio before and after a small change to judge whether reducing waiting improved flow.
- Recovery time by incident severity. Explain what MTTR measures, what a high value may signal, and why critical incidents should be tracked separately.
- Change failure rate guardrail. Explain what change failure rate counts and why it must be read alongside deployment frequency.
- Metrics as diagnostic signals. Explain how to use the metrics together as learning signals and map a change back through the value stream.
- Vanity metrics versus value. Distinguish activity counts from measures of value delivery using examples from the section.
- Metric targets and gaming. Explain how targeting a metric can improve its number while making the delivery system worse, without treating gaming as a personal failing.
- Paired metric guardrails. Apply the section's system-improvement question and explain how a guardrail distinguishes healthy gains from apparent ones.
- Kanban as flow diagnosis. Distinguish a board that tracks ticket status from a visual management system that shows how work is flowing. Explain how Toyota’s live, visible boards illustrate the broader purpose of kanban.
- Four visible flow signals. Describe how column depth, WIP limits, ageing items, and blocked indicators let a team diagnose problems on its existing board before building a separate dashboard.
- Snapshots and flow trends. Distinguish what a Kanban board shows from what a flow dashboard shows, and explain why a team needs both.
- Lead and cycle time displays. Explain how to display lead time and cycle time so their signals are not hidden by averages.
- Dashboard conditions for improvement. Explain why dashboard audience, freshness, simplicity, and team-level focus determine whether it supports improvement.
- Trends over isolated snapshots. Explain why a dashboard should show metrics over time rather than only their current values, with enough history to reveal patterns and abnormalities.
- Distributions and stalled work. Explain how a distribution can expose groups of fast and slow work hidden by an average, and describe the next step when such a split appears.
- Visible effects of improvements. Describe how a dashboard should show whether an improvement experiment changes flow, using one of the section's examples.
- Delivery-time calculations. Explain how to calculate lead time, cycle time, wait time, and activity ratio for each sampled item.
- Signals of delivery friction. Describe how to inspect the measurements and work history for queues and repeated delays.
- Measured improvement experiment. Explain how to choose one improvement based on a detected delay and assess whether it helped.
What it builds on
- Understanding flow, from idea to production (chapter 6) · The idea of work flowing from an idea to production.
- The eight wastes in software development (chapter 2) · The waste categories used to describe waiting, handoffs, and overprocessing.
- Designing for intervention: Enabling the software Andon (chapter 4) · The idea of making problems visible through a software andon signal.
- Gemba and Kaizen in software development (chapter 2) · The practice of going to the gemba to observe work.
- Identifying and avoiding bottlenecks (chapter 6) · The idea of a bottleneck limiting flow.
- Flow metrics (chapter 5) · The earlier introduction of flow metrics.
- Reducing batch sizes for faster feedback (chapter 6) · Reducing batch sizes for faster feedback.
- Limiting work in progress (chapter 6) · Work-in-progress limits.
Section by section
Introduction
The chapter introduction frames metrics as a way to expose hidden system problems and judge whether measured improvement is real.
- Visibility into system problems: Explain why teams measure software delivery: to make otherwise invisible stalls, queues, and system-created drag visible in data they can act on.
Why it matters: Seeing where the system slows work gives a team something concrete to improve. - Real improvement versus metric gaming: Distinguish a useful metric from a harmful one by asking whether improving its number actually improves the system, rather than rewarding busywork or dashboard gaming.
Why it matters: A better-looking dashboard is not a win if the work itself has not improved.
Key terms: metrics, value stalls, queues, wrong metrics, busywork, game dashboards, improving outcomes
9.1 Why measure? Making problems visible
This section shows why measuring end-to-end delivery time, rather than local activity, makes hidden delays in software delivery visible.
- System Versus Station Measurement: Explain why a team's coding speed or deployment frequency can give a misleading picture of delivery. Describe why the customer-facing time from request to production is the more useful measure here.
Why it matters: It helps you notice delivery problems that busy teams and frequent deployments can conceal. - Hidden Time in Delivery: Describe what the fintech team's measurements revealed about its 23-day lead time and why the section treats the gap between lead time and activity time as a system problem.
Why it matters: Separating active work from elapsed time directs attention to hidden process delays instead of blaming developers.
Key terms: system instead of the station, lead time, activity time, activity ratio, TIMWOODS-centric waste
9.1.1 Measurement for learning versus ranking
This section explains why team metrics should reveal system problems for improvement rather than rank individuals, which can hide problems and encourage gaming.
- System patterns behind metrics: Explain why a developer's metric may reflect the delivery system rather than individual performance, using the on-call cycle-time example.
Why it matters: Looking for the system pattern gives a team something it can improve instead of someone to blame. - Fear and problem signals: Explain how attaching reported problems or metrics to individual performance judgments affects whether developers make system problems visible.
Why it matters: A team cannot improve problems that people have learned to conceal. - Management by means versus results: Distinguish using numerical targets to judge people from using shared metrics to understand and improve the processes producing results.
Why it matters: The purpose assigned to a metric determines whether it helps improve delivery or encourages people to defend their numbers.
Key terms: variation, cycle time, system pattern, driving out fear, andon cord, information flow, management by results, management by means, measure the system, not the person
9.1.2 The optimization trap
This section shows how targeting a metric can undermine the system it measures, and how to use metrics to investigate and improve the process instead.
- Diagnostic signals versus targets: Explain why using a metric as a target can make the number improve while reliable delivery gets worse, using the build-time example.
Why it matters: A better-looking number is not evidence of improvement if the delivery system has become less reliable. - Process-based metric improvement: Describe how a team should respond to an unfavorable metric and why a numerical target needs a method for changing the process.
Why it matters: Investigating the work helps a team improve the cause of a bad result rather than conceal the result. - Bottleneck-limited local improvement: Explain why speeding up one station beyond the bottleneck does not improve overall throughput.
Why it matters: A local improvement can make one station look faster while leaving the system no better off.
Key terms: Goodhart's Law, diagnostic signal, gamification, change failure rate, gemba, numerical target, process, production boards, local optimization, bottleneck
9.2 Lean metrics for developers
This section reframes four flow metrics as ways to observe how everyday developer work moves or stalls.
- Developer view of flow metrics: Explain that lead time, cycle time, activity time, and activity ratio measure how value moves through a system, and that developers can apply this flow perspective to tickets, PRs, and the deploy pipeline.
Why it matters: This framing connects flow measurements to work developers can observe directly.
Key terms: flow metrics, lead time, cycle time, activity time, activity ratio
9.2.1 Lead time
Defines lead time as the full request-to-delivery journey and shows how to measure and interpret it to find workflow delays without turning the metric into a target.
- End-to-end lead time: Explain what lead time measures and why fast coding alone may not mean fast delivery.
Why it matters: This measure reflects how long customers actually wait for a result. - Practical lead time collection: Describe how a team can obtain lead-time data with an existing platform or a simple script, and why lack of a platform need not prevent measurement.
Why it matters: A team can start measuring without first committing to another platform. - Typical and extreme lead times: Distinguish what the median and p90 reveal about completed tickets, and explain why an average can misrepresent typical flow when there is a large outlier.
Why it matters: Separating typical delays from exceptional ones helps the team investigate the right problem. - Control chart variation signals: Explain how a lead-time control chart distinguishes ordinary variation from work items worth investigating.
Why it matters: The chart helps the team notice unusual tickets without reacting to every normal fluctuation. - Outlier investigation over targets: Describe the section's starting approach to measuring lead time, investigating slow outliers, and avoiding lead-time targets.
Why it matters: Finding the delayed segment directs improvement toward the system rather than toward a number people can manipulate.
Key terms: lead time, request to delivery, system friction, engineering intelligence platforms, observability tools, issue tracker events, commit timestamps, deploy logs, median, p50
9.2.2 Cycle time
This section defines cycle time and shows how to use it to locate delays, evaluate improvements, and learn without ranking developers.
- Cycle time boundaries: Explain what cycle time measures, how it differs from lead time, and why a team must agree on when work starts and counts as complete.
Why it matters: Consistent boundaries make cycle times meaningful across work items. - Batch size and cycle time: Explain why larger work items can have longer cycle times even when developers code at the same speed.
Why it matters: A long cycle time does not, by itself, show that someone codes slowly. - Prework queue time gap: Explain what the gap between lead time and cycle time reveals and what kind of improvement a large gap suggests.
Why it matters: The comparison prevents a team from trying to code faster when work is mostly waiting to start. - Control charts and phase diagnosis: Describe how comparing lead-time and cycle-time charts, then decomposing unusually long cycle times, helps locate a constraint.
Why it matters: A single duration cannot tell you whether to address coding, review waiting, or an earlier queue. - Work type and experiment comparisons: Describe how comparing cycle times across work item types and before and after an improvement experiment avoids misleading averages and tests whether the change addressed a constraint.
Why it matters: These comparisons help a team see both hidden bottlenecks and whether its changes helped. - Learning rather than developer ranking: Justify using cycle-time charts to investigate work items rather than using individual cycle-time targets to rank developers.
Why it matters: How a team uses the measure determines whether it uncovers delays or distorts behavior.
Key terms: cycle time, lead time, Definition of Ready (DoR), Definition of Done (DoD), batch size, queue time, WIP limits, control chart, control limits, decomposition
9.2.3 Activity time
This section uses activity time and activity ratio to reveal hidden waiting, diagnose where flow slows, and test improvements without treating the measures as productivity targets.
- Active work versus waiting: Distinguish activity time from elapsed time for a work item, giving examples of active progress and passive waiting.
Why it matters: Separating progress from waiting makes delays visible that elapsed time alone cannot explain. - Hidden queues within stations: Explain how comparing activity time with elapsed time at a process station exposes waiting that a value stream map may not show.
Why it matters: This comparison helps you investigate a slow station instead of assuming its actual work takes too long. - Activity ratios and improvement experiments: Describe how to use activity ratio before and after a small change to judge whether reducing waiting improved flow.
Why it matters: A before-and-after ratio tests whether a process change reduced waiting rather than merely asking people to work harder. - Diagnostic use of activity data: Explain how a rough activity log and a breakdown by activity type can reveal possible causes of waste, while avoiding activity time as a productivity score.
Why it matters: Used diagnostically, the measure points toward process problems without rewarding shortcuts that hide defects.
Key terms: activity time, elapsed time, value stream map, queue time, activity ratio, review windows, value stream, queue, activity type, diagnostic lens
9.2.4 Beyond the fundamentals
Extends flow measurement into production and shows how to use operational metrics to locate problems without turning them into scorecards.
- Recovery time by incident severity: Explain what MTTR measures, what a high value may signal, and why critical incidents should be tracked separately.
Why it matters: A recovery average is useful only if it reveals the failures that matter most. - Deployment frequency and batch size: Describe what deployment frequency measures and how smaller batches relate to frequent, reliable releases.
Why it matters: Release counts make more sense when read as evidence of how work is batched and delivered. - Frequency and stability quadrants: Distinguish the four combinations of deployment frequency and stability and what each suggests about flow and quality.
Why it matters: The same release frequency can describe healthy flow or repeated production breakage. - Change failure rate guardrail: Explain what change failure rate counts and why it must be read alongside deployment frequency.
Why it matters: Shipping more often is an improvement only if changes remain reliable. - Metrics as diagnostic signals: Explain how to use the metrics together as learning signals and map a change back through the value stream.
Why it matters: A metric is most useful when it starts an investigation rather than ends one or becomes a target to game.
Key terms: MTTR, Mean Time to Restore, critical incidents, deployment frequency, batch size, continuous delivery, smooth flow, bottlenecked, chaotic, stalled
9.3 Avoiding vanity metrics and metric gamification
The section shows how activity-based metrics can reward behavior that worsens delivery, and how paired system metrics and gradual replacement make measurement more useful.
- Vanity metrics versus value: Distinguish activity counts from measures of value delivery using examples from the section.
Why it matters: An impressive number is only useful if it tells the team something about delivered value. - Metric targets and gaming: Explain how targeting a metric can improve its number while making the delivery system worse, without treating gaming as a personal failing.
Why it matters: A better dashboard can conceal a slower or less reliable delivery system. - Paired metric guardrails: Apply the section's system-improvement question and explain how a guardrail distinguishes healthy gains from apparent ones.
Why it matters: A guardrail helps the team spot apparent speed gains that come with worse outcomes. - Gradual replacement of vanity metrics: Describe how a team can replace an established vanity metric with a more useful system-level signal.
Why it matters: A visible improvement the team experiences can earn more trust than a mandate to stop using a familiar measure.
Key terms: vanity metric, value delivery, local measurements, system measurements, metric gamification, cycle time, coherent change, system improvement, paired metrics, guardrail
9.4 Visual dashboards for continuous improvement
This section reframes a Kanban board as a live visual tool for diagnosing flow, not merely tracking ticket status.
- Kanban as flow diagnosis: Distinguish a board that tracks ticket status from a visual management system that shows how work is flowing. Explain how Toyota’s live, visible boards illustrate the broader purpose of kanban.
Why it matters: Reading the board as a picture of flow helps the team spot problems rather than just find its next ticket. - Four visible flow signals: Describe how column depth, WIP limits, ageing items, and blocked indicators let a team diagnose problems on its existing board before building a separate dashboard.
Why it matters: These signals turn a board the team already uses into a way to notice where flow needs attention.
Key terms: kanban, visual management system, inventory tracking, flow diagnosis, pull-based system, column depths, queues, WIP limits, ageing items, blocked indicators
9.4.1 From flow signals to flow dashboards
This section shows how to turn flow metrics into a useful team dashboard that reveals trends without becoming stale, overwhelming, or a tool for ranking people.
- Snapshots and flow trends: Distinguish what a Kanban board shows from what a flow dashboard shows, and explain why a team needs both.
Why it matters: A current pileup calls for attention, but its trend helps the team understand whether the problem is growing. - Activity recovery and release signals: Describe how activity ratio, MTTR trend, deployment frequency, and change failure rate give different signals about system health.
Why it matters: These signals keep a busy or frequently deploying team from overlooking waste, recovery difficulty, or failures. - Lead and cycle time displays: Explain how to display lead time and cycle time so their signals are not hidden by averages.
Why it matters: The way a metric is displayed determines whether the team can see a bottleneck or a pattern hidden by an average. - Dashboard conditions for improvement: Explain why dashboard audience, freshness, simplicity, and team-level focus determine whether it supports improvement.
Why it matters: A dashboard is useful only if the team can trust, read, and act on it rather than feel ranked by it.
Key terms: Kanban board, flow dashboard, visual management, activity ratio, MTTR trend, deployment frequency, change failure rate, lead time trend, median, cycle time distribution
9.4.2 Making it work in practice
This section explains how to make flow dashboards useful for continuous improvement by showing patterns, revealing stalled work, and making the effects of changes visible.
- Trends over isolated snapshots: Explain why a dashboard should show metrics over time rather than only their current values, with enough history to reveal patterns and abnormalities.
Why it matters: A trend helps the team tell whether a number signals progress or a developing problem. - Distributions and stalled work: Explain how a distribution can expose groups of fast and slow work hidden by an average, and describe the next step when such a split appears.
Why it matters: Seeing which work gets stuck gives the team a more useful starting point than a reassuring average. - Visible effects of improvements: Describe how a dashboard should show whether an improvement experiment changes flow, using one of the section's examples.
Why it matters: Showing what changed helps the team judge whether an intervention improved the system rather than just a single measure. - Accessible and maintained visibility: Justify starting with a simple, visible dashboard that the team actually uses rather than waiting for a sophisticated platform.
Why it matters: An accessible measure the team updates and sees can prompt action when an elaborate but ignored dashboard cannot. - Metrics matched to current focus: Describe when to add or remove dashboard metrics as a team's improvement focus changes.
Why it matters: Rotating metrics keeps the dashboard focused on questions the team is currently trying to answer. - System level AI measurement: Explain why AI-assisted coding should be judged by its effect on overall flow and quality rather than coding speed alone.
Why it matters: Faster code generation may leave the delays and failures that determine delivery performance untouched.
Key terms: trends, snapshots, time series data, baseline, distributions, averages, bimodal distribution, classification, improvement experiment, activity ratio
9.5 Exercise: Calculate your current lead time
This exercise uses a representative sample of completed work to calculate delivery times, locate delays, and test one measurable improvement.
- Representative completed-work sample: Describe how to select work items and collect the data needed for the exercise.
Why it matters: A representative sample makes the resulting picture of delivery less misleading. - Delivery-time calculations: Explain how to calculate lead time, cycle time, wait time, and activity ratio for each sampled item.
Why it matters: These calculations separate the customer's total wait from time after work began and from hands-on effort. - Signals of delivery friction: Describe how to inspect the measurements and work history for queues and repeated delays.
Why it matters: Looking beyond one headline number helps locate a delay worth addressing. - Measured improvement experiment: Explain how to choose one improvement based on a detected delay and assess whether it helped.
Why it matters: A before-and-after experiment connects a specific change to a real improvement in delivery.
Key terms: completed changes, representative sample, active working time, lead time, cycle time, wait time, activity ratio, review queue, CI, rework
9.6 Summary
Where this is used later
- (chapter intro) (chapter 10) · Measurement as a way to make problems visible.
- Why retrospectives fail (chapter 10) · Using measurement to make problems visible.
- Developer-led Kaizen (chapter 10) · Measurement makes problems visible.
- Finding the system bottleneck (chapter 11) · Provides an example of why the system bottleneck matters more than the speed of an individual station.
- From mindset to method (chapter 10) · Selecting metrics to describe a friction point.
- Development velocity versus operational stability (chapter 12) · Development flow metrics such as lead time and cycle time.
- What Kaizen actually means (chapter 10) · Provides a measurement-led starting point for continuous improvement.
- Demonstrating value through small wins (chapter 15) · demonstrating value through a small, measurable improvement
Concepts explained in this chapter
- lead timeThe elapsed time between the start of work and delivery of its result.
- activity ratioThe proportion of elapsed work time spent on productive activity rather than waiting or other non-progressing periods.
- change failure rateThe proportion of software changes that cause a failure requiring remediation, rollback, or another corrective response.
- mean time to resolutionThe average time required to restore a system or service after an incident.
- control chartA statistical process-control chart that plots process data over time against a center line and control limits to distinguish common-cause variation from unusual variation.
- CPU timeThe time spent actively progressing a change through activities such as coding, testing, fixing, and reviewing.
- cycle timeThe elapsed time required for work to move through a production or delivery process.
- visual managementA system that makes the current state of work visible to the people responsible for it.
- vanity metricA measure that can appear to indicate success without showing whether the system’s outcomes actually improve.
- metricA metric is a quantitative measure of a property or outcome of a process, system, or activity.
- metric gamingThe behavior of optimizing a measured number to meet its target without improving the underlying outcome.
- paired metricA measure considered alongside another measure that guards against harmful changes to the underlying system.
- queue timeThe elapsed time a work item spends waiting rather than being actively processed.
- process variationFluctuation in a process’s measured outcomes over time or across operating conditions.
- wasteWork that creates no customer value and can be eliminated without harming delivery or its outcomes.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
