liveBook · Manning

Offline metric

As explained in AI Model Evaluation

An offline metric is a quantitative measure of model or algorithm behavior computed from historical or otherwise non-live evaluation data.

An offline metric is a quantitative measure of model behavior computed without exposing a change to users in a live environment. It helps identify promising candidates and potential regressions before user-facing experimentation, but it does not establish the change’s real-world impact. section 1.3 section 6.0

Choose metrics for the product question, the model’s role, and the available data. A single number rarely captures the whole experience: a recommender can rank relevant titles well while repeatedly showing a narrow selection. Pair relevance measures with diversity or coverage when those outcomes matter. section 1.3.5

Common choices depend on what the system must do:

  • For classification, use measures such as precision, recall, or F1; for ranking and retrieval, consider Precision@K, Recall@K, or NDCG@K. section 1.3.4 section 1.3.5
  • For generated answers, assess qualities such as relevance and groundedness. For agents, consider task completion, constraint adherence, and tool-call accuracy. section 1.3.4 section 1.3.5
  • An LLM judge is a scoring method, not itself a metric: it might score groundedness against a rubric, and its judgments need validation and calibration. section 1.3.4

Precision at five for recommendations

A streaming app recommends five shows. If historical watch behavior or labeled preferences identify three as relevant, Precision@5 is 3/5, or 60%. This measures relevance among the displayed top five; it does not tell you whether the selection is diverse or whether users will enjoy the experience. section 1.3.5

The data behind a metric matters as much as the formula. Historical events and outcomes support backtesting; validation data helps tune models, while a separate holdout supports a more independent final assessment. Check whether the data represents current users and product conditions: outdated behavior, previous-model exposure bias, or incomplete logging can mislead. Synthetic data can cover rare cases, but should be validated against realistic production patterns. section 1.3.3

Use offline results to guide the next decision, not to replace it. A/B tests reveal how models interact with real users, interfaces, latency constraints, and feedback loops. If online and offline results disagree, inspect segments, logs, inputs, exposure, and metric alignment rather than assuming one cause. section 6.0 section 6.5.5

Carry known offline risks into online monitoring. For each important measure, define a live watchpoint, computation logic, and alert threshold. An online click rate for the top ten results is only a proxy for offline Precision@10: clicks also depend on presentation, position, intent, and exposure. Track the relationship over time rather than treating the measures as identical. section 7.6 section 7.6.2

Table 7.2 Example: Metric Mirror Table for Recommendation Model A/B Test
Offline Metric
Online Watch point
Thresholds  
Target Value
Diversity index
1−∑pi2 computed daily from top-N categories in live logs
Drop > 0.05 vs. control for ≥ 3 days
≥ 0.70
Latency p95
p95(model_response_time_ms) aggregated per day
> 400ms
≤ 350ms
Bias score by segment
Mean(predicted_score[group]) difference between any two groups (absolute value)
Gap > 0.10 for any demographic segment
≤ 0.05
Precision@10
Top-10 click-rate proxy: clicks on items shown in the top 10 divided by top-10 impressions, aggregated daily
Drop > 0.02 vs. control
Match or exceed control
Fairness gap
min(segment_score)/max(segment_score)
Ratio < 0.85
≥ 0.90

Compare offline measures with live watchpoints and example alert thresholds for a recommendation experiment. section 7.6.2

How AI Model Evaluation covers it

How deep it goes
Explains metric selection, data preparation, interpretation, and online monitoring, including a worked Precision@5 calculation.
Examples
Calculates Precision@5 for recommended shows and maps recommendation metrics to live watchpoints and thresholds.
What you can do after
Evaluate and understand model changes by selecting product-aligned metrics, checking data representativeness, and comparing offline signals with online outcomes.
How it connects
Connects offline evaluation with ranking, retrieval, groundedness, A/B testing, and online monitoring.

Also explained in

Compare how 2 books explain offline metric →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.