A proxy metric is a measurable stand-in for a quality or outcome that is difficult or slow to measure directly. It can provide a leading signal for daily decisions, but an improvement in the proxy does not by itself establish an improvement in the outcome. section 8.5.1 section 8.5
Use proxies within a metric hierarchy. Outcome metrics define success, proxies provide more immediate signals, and diagnostic metrics help explain model behavior. Guardrails and diagnostics can reveal harms that a headline proxy misses. section 8.5.1 section 8.5.3
Movie recommendations
A recommender can use 30-day retention as a lagging outcome, recommendation click-through rate and watch-time per session as leading proxies, and genre diversity and cohort fairness gaps as diagnostics. A 3% increase in click-through rate looks less convincing if genre diversity falls by 40% and new users’ watch-time drops. section 8.5.1
Interpret behavioral signals with care. A click may reflect prominent placement rather than preference. Likewise, top-10 click rate is not the same as offline Precision@10: clicks depend on position, presentation, intent, and exposure, while offline precision uses labels or relevance judgments. section 2.2.1 section 7.6.2
Table 7.2 Example: Metric Mirror Table for Recommendation Model A/B Test
|
Offline Metric
|
Online Watch point
|
Thresholds
|
Target Value
|
|
Diversity index
|
1−∑pi2 computed daily from top-N categories in live logs
|
Drop > 0.05 vs. control for ≥ 3 days
|
≥ 0.70
|
|
Latency p95
|
p95(model_response_time_ms) aggregated per day
|
> 400ms
|
≤ 350ms
|
|
Bias score by segment
|
Mean(predicted_score[group]) difference between any two groups (absolute value)
|
Gap > 0.10 for any demographic segment
|
≤ 0.05
|
|
Precision@10
|
Top-10 click-rate proxy: clicks on items shown in the top 10 divided by top-10 impressions, aggregated daily
|
Drop > 0.02 vs. control
|
Match or exceed control
|
|
Fairness gap
|
min(segment_score)/max(segment_score)
|
Ratio < 0.85
|
≥ 0.90
|
The metric mirror table pairs offline Precision@10 with a daily top-10 click-rate watchpoint, making the proxy relationship explicit alongside thresholds and other checks. section 7.6.2
Keep proxies useful as the product and model change:
- Check whether the proxy still tracks the desired outcome using historical correlation studies; update it when the relationship weakens. section 8.5.2
- Version metric definitions and refresh diagnostics as risks change. For an LLM-based recommender, persuasive explanations may drive clicks without improving satisfaction, so factuality, coherence, novelty, and hallucination rates also matter. section 8.5.2
- Combine offline, online, human, and LLM-as-a-judge evaluations rather than relying on online metrics alone. section 8.0







