liveBook · Manning

Proxy metric

As explained in AI Model Evaluation

A proxy metric is an indirect measure whose value is intended to indicate a target quality or outcome that is difficult to measure directly.

A proxy metric is a measurable stand-in for a quality or outcome that is difficult or slow to measure directly. It can provide a leading signal for daily decisions, but an improvement in the proxy does not by itself establish an improvement in the outcome. section 8.5.1 section 8.5

Use proxies within a metric hierarchy. Outcome metrics define success, proxies provide more immediate signals, and diagnostic metrics help explain model behavior. Guardrails and diagnostics can reveal harms that a headline proxy misses. section 8.5.1 section 8.5.3

Movie recommendations

A recommender can use 30-day retention as a lagging outcome, recommendation click-through rate and watch-time per session as leading proxies, and genre diversity and cohort fairness gaps as diagnostics. A 3% increase in click-through rate looks less convincing if genre diversity falls by 40% and new users’ watch-time drops. section 8.5.1

Interpret behavioral signals with care. A click may reflect prominent placement rather than preference. Likewise, top-10 click rate is not the same as offline Precision@10: clicks depend on position, presentation, intent, and exposure, while offline precision uses labels or relevance judgments. section 2.2.1 section 7.6.2

Table 7.2 Example: Metric Mirror Table for Recommendation Model A/B Test
Offline Metric
Online Watch point
Thresholds  
Target Value
Diversity index
1−∑pi2 computed daily from top-N categories in live logs
Drop > 0.05 vs. control for ≥ 3 days
≥ 0.70
Latency p95
p95(model_response_time_ms) aggregated per day
> 400ms
≤ 350ms
Bias score by segment
Mean(predicted_score[group]) difference between any two groups (absolute value)
Gap > 0.10 for any demographic segment
≤ 0.05
Precision@10
Top-10 click-rate proxy: clicks on items shown in the top 10 divided by top-10 impressions, aggregated daily
Drop > 0.02 vs. control
Match or exceed control
Fairness gap
min(segment_score)/max(segment_score)
Ratio < 0.85
≥ 0.90

The metric mirror table pairs offline Precision@10 with a daily top-10 click-rate watchpoint, making the proxy relationship explicit alongside thresholds and other checks. section 7.6.2

Keep proxies useful as the product and model change:

  • Check whether the proxy still tracks the desired outcome using historical correlation studies; update it when the relationship weakens. section 8.5.2
  • Version metric definitions and refresh diagnostics as risks change. For an LLM-based recommender, persuasive explanations may drive clicks without improving satisfaction, so factuality, coherence, novelty, and hallucination rates also matter. section 8.5.2
  • Combine offline, online, human, and LLM-as-a-judge evaluations rather than relying on online metrics alone. section 8.0

How AI Model Evaluation covers it

How deep it goes
Defines metric hierarchy layers and explains proxy failure, validation, and revision as models and products evolve.
Examples
Uses movie recommendation clicks, watch-time, retention, genre diversity, and cohort fairness to illustrate a metric hierarchy.
What you can do after
Evaluate and revise proxies against outcomes, diagnostics, and historical correlations rather than treating a single engagement metric as success.
How it connects
Links proxy metrics to outcome metrics, diagnostic metrics, offline evaluation, online evaluation, and guardrail metrics.

Also explained in

Compare how 8 books explain proxy metric →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.