liveBook · Manning

Metric hierarchy

As explained in AI Model Evaluation

A metric hierarchy is an ordered structure of metrics that establishes their relative priority when evaluating trade-offs.

A metric hierarchy orders evaluation signals so teams can resolve trade-offs between model behavior, product goals, and user impact. It prevents a convenient proxy from standing in for real success. section 3.4.6 section 8.5

Build the hierarchy in three layers:

  • Outcome metrics measure durable product value, such as retention or monthly active users. They define success but often lag and cannot guide daily iteration. section 8.5.1
  • Proxy metrics provide faster, measurable stand-ins for outcomes, such as recommendation CTR, watch time, conversation length, or response acceptance rate. Treat them as signals, not proof. section 8.5.1
  • Diagnostic metrics explain model behavior and expose risks, including recommendation diversity, cohort fairness gaps, latency, and hallucination rates. section 8.5.1

Movie recommender

Set 30-day retention as the outcome, CTR and average watch time as proxies, and genre diversity plus fairness gaps across user cohorts as diagnostics. A 3% CTR increase is not sufficient evidence of success if genre diversity drops 40% or new users’ watch time declines. section 8.5.1

Use offline evaluation to surface diagnostic weaknesses, then use online evaluation to test whether proxy and outcome metrics move together. Revisit the hierarchy as products, models, and strategic goals change. section 8.5 section 8.5.3

How AI Model Evaluation covers it

How deep it goes
Defines three metric layers and their roles, then stops short of prescribing a universal measurement formula.
Examples
Uses movie recommendation retention, CTR, watch time, genre diversity, and cohort fairness as concrete measures.
What it assumes
Assumes awareness of product goals and model behavior, while distinguishing their evaluation roles explicitly.
What you can do after
Evaluate model changes by separating durable outcomes, fast proxies, and diagnostics that explain regressions or hidden harms.
How it connects
Connects outcome metrics, proxy metrics, diagnostic metrics, offline evaluation, online evaluation, and product strategy.

Also explained in

Compare how 2 books explain metric hierarchy →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.