liveBook · Manning

Vanity metric

As explained in Build High Value Software Systems

A measure that can appear to indicate success without showing whether the system’s outcomes actually improve.

A vanity metric rises without showing whether value delivery improves. It measures activity at one point in the workflow, not whether value moves through the system. section 9.3

Engineering measures can look impressive while rewarding activity over outcomes:

  • Lines of code reward bloat and penalize simplification. section 9.3
  • Story points and tickets closed measure estimated effort, not delivered value. section 9.3
  • Pull requests and commits can reward fragmentation or frequency over coherent work. section 9.3
  • Test counts and coverage percentages can grow without meaningful failure-mode coverage. section 9.3
  • AI prompts or tokens count interactions, not whether their output was useful or correct. section 9.3

A useful check is: if this number improves, does the system improve too? Once a metric becomes a target, people may rationally optimize the number without improving outcomes. section 9.3 section 9.0

Listing 9.2 The gap between metric improvement and system improvement
# What the metric sees
pr_cycle_times = [1.2, 0.8, 1.5, 0.9, 1.1]  # days per PR
avg_cycle = sum(pr_cycle_times) / len(pr_cycle_times)
print(f"Average PR cycle time: {avg_cycle:.1f} days")  # 1.1 days, great!
 
# What actually happened
feature_lead_time = 14  # days from request to production
prs_per_feature = 5     # artificially split
total_review_overhead = prs_per_feature * 0.5  # hours per PR for context switching
 
# The metric improved. The system didn't.
# Feature lead time is unchanged. Review overhead increased.

A lower average PR cycle time looks like progress, but feature lead time stays unchanged while review overhead rises. section 9.3

How Build High Value Software Systems covers it

How deep it goes
Defines vanity metrics by their failure to track value, then explains examples, local-versus-system measurement, and target-driven gaming.
Examples
Uses lines of code, story points, merged pull requests, test counts, commits, AI prompts, and PR cycle time as illustrations.
What you can do after
Evaluate engineering metrics by checking whether improvement reflects better system outcomes, then pair speed measures with quality guardrails.
How it connects
Connects metric gaming with lead time, cycle time, deployment frequency, change failure rate, and system-level measurement.

Also explained in

Compare how 3 books explain vanity metric →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build High Value Software Systems. The section links point to the parts of the book it draws on.