Suppose a sales forecast looks more accurate on validation data. Will it actually reduce losses from empty shelves or excess stock? An offline metric measures model performance before deployment, using data available for testing. It makes comparisons quick and reproducible, but it is not automatically a measure of business success. section 5.2.2 section 12.4.1
Choose offline metrics as proxies for the outcomes you care about. A useful proxy should predict the direction or size of change in an online metric. That relationship may be weak: better prediction scores do not necessarily mean better user or business results. Compare past offline results with corresponding online tests when possible, then use an A/B test to check the effect of a new version in production. section 5.2.2 section 12.1.1
Table 5.1 Examples of offline and online metrics
|
Offline metrics
|
Online metrics
|
|---|---|
|
Recall at given specificity for spam message classification
|
Number of user complaints about spam messages
|
|
Quantiles of 1.5, 25, 50, 75, 95, and 99
|
Value of expired items, total sales
|
|
Mean reciprocal rank, normalized discounted cumulative gain
|
Click-through rate on search engine result page
|
Compare each offline measure with the live outcome it is meant to predict: spam-classification recall with complaints, forecast error quantiles with stock-related sales and losses, and search-ranking scores with click-through rate. section 5.2.2
Match the metric to the decision and the data it needs:
- For image-search results, category diversity can be checked offline without additional labels; human evaluation needs judgments that take time and effort to collect. section 5.3.2
- Mean reciprocal rank and normalized discounted cumulative gain can proxy for search outcomes, but require relevant-result lists. A list suitable for predicting clicks may not represent conversions. section 5.3.2
- For sales forecasts, inspect error quantiles, percentage errors, and bias by product category—not just an overall score. Check whether findings recur across validation folds, seasons, and regions. section 12.4.1
Treat an offline improvement as a reason to investigate, not as a claimed revenue gain. Prior A/B tests can help estimate the likely effect of a forecast improvement; a new live test checks that estimate before a rollout decision. section 12.4.1 section 12.5
