A metric hierarchy orders evaluation signals so teams can resolve trade-offs between model behavior, product goals, and user impact. It prevents a convenient proxy from standing in for real success. section 3.4.6 section 8.5
Build the hierarchy in three layers:
- Outcome metrics measure durable product value, such as retention or monthly active users. They define success but often lag and cannot guide daily iteration. section 8.5.1
- Proxy metrics provide faster, measurable stand-ins for outcomes, such as recommendation CTR, watch time, conversation length, or response acceptance rate. Treat them as signals, not proof. section 8.5.1
- Diagnostic metrics explain model behavior and expose risks, including recommendation diversity, cohort fairness gaps, latency, and hallucination rates. section 8.5.1
Movie recommender
Set 30-day retention as the outcome, CTR and average watch time as proxies, and genre diversity plus fairness gaps across user cohorts as diagnostics. A 3% CTR increase is not sufficient evidence of success if genre diversity drops 40% or new users’ watch time declines. section 8.5.1
Use offline evaluation to surface diagnostic weaknesses, then use online evaluation to test whether proxy and outcome metrics move together. Revisit the hierarchy as products, models, and strategic goals change. section 8.5 section 8.5.3

