An offline metric is a quantitative measure of model behavior computed without exposing a change to users in a live environment. It helps identify promising candidates and potential regressions before user-facing experimentation, but it does not establish the change’s real-world impact. section 1.3 section 6.0
Choose metrics for the product question, the model’s role, and the available data. A single number rarely captures the whole experience: a recommender can rank relevant titles well while repeatedly showing a narrow selection. Pair relevance measures with diversity or coverage when those outcomes matter. section 1.3.5
Common choices depend on what the system must do:
- For classification, use measures such as precision, recall, or F1; for ranking and retrieval, consider Precision@K, Recall@K, or NDCG@K. section 1.3.4 section 1.3.5
- For generated answers, assess qualities such as relevance and groundedness. For agents, consider task completion, constraint adherence, and tool-call accuracy. section 1.3.4 section 1.3.5
- An LLM judge is a scoring method, not itself a metric: it might score groundedness against a rubric, and its judgments need validation and calibration. section 1.3.4
Precision at five for recommendations
A streaming app recommends five shows. If historical watch behavior or labeled preferences identify three as relevant, Precision@5 is 3/5, or 60%. This measures relevance among the displayed top five; it does not tell you whether the selection is diverse or whether users will enjoy the experience. section 1.3.5
The data behind a metric matters as much as the formula. Historical events and outcomes support backtesting; validation data helps tune models, while a separate holdout supports a more independent final assessment. Check whether the data represents current users and product conditions: outdated behavior, previous-model exposure bias, or incomplete logging can mislead. Synthetic data can cover rare cases, but should be validated against realistic production patterns. section 1.3.3
Use offline results to guide the next decision, not to replace it. A/B tests reveal how models interact with real users, interfaces, latency constraints, and feedback loops. If online and offline results disagree, inspect segments, logs, inputs, exposure, and metric alignment rather than assuming one cause. section 6.0 section 6.5.5
Carry known offline risks into online monitoring. For each important measure, define a live watchpoint, computation logic, and alert threshold. An online click rate for the top ten results is only a proxy for offline Precision@10: clicks also depend on presentation, position, intent, and exposure. Track the relationship over time rather than treating the measures as identical. section 7.6 section 7.6.2
Table 7.2 Example: Metric Mirror Table for Recommendation Model A/B Test
|
Offline Metric
|
Online Watch point
|
Thresholds
|
Target Value
|
|
Diversity index
|
1−∑pi2 computed daily from top-N categories in live logs
|
Drop > 0.05 vs. control for ≥ 3 days
|
≥ 0.70
|
|
Latency p95
|
p95(model_response_time_ms) aggregated per day
|
> 400ms
|
≤ 350ms
|
|
Bias score by segment
|
Mean(predicted_score[group]) difference between any two groups (absolute value)
|
Gap > 0.10 for any demographic segment
|
≤ 0.05
|
|
Precision@10
|
Top-10 click-rate proxy: clicks on items shown in the top 10 divided by top-10 impressions, aggregated daily
|
Drop > 0.02 vs. control
|
Match or exceed control
|
|
Fairness gap
|
min(segment_score)/max(segment_score)
|
Ratio < 0.85
|
≥ 0.90
|
Compare offline measures with live watchpoints and example alert thresholds for a recommendation experiment. section 7.6.2

