An online metric measures a system after deployment, when people use its output in a real business setting. It may be the goal that matters most—such as sales—but it cannot be calculated from model predictions alone. Offline metrics are quicker to calculate during development; their value as proxies depends on how well they predict the online result. section 5.2.2
From search ranking to purchases
For an image-search system, click-through rate and conversion rate are online metrics: both require live user behavior. Mean reciprocal rank and normalized discounted cumulative gain can be calculated offline from lists of relevant results. Improving those ranking scores may be promising, but it does not establish that users will click or buy more. section 5.3.2 section 12.1.1
Choose the business goal first, then identify the online metric and any offline proxies that support faster iteration. A recommender might aim to increase gross merchandise value while tracking an offline measure before launch. Do not assume a better proxy guarantees a better business outcome: compare offline results with a history of online tests where possible. section 5.2.2 section 12.1.1
Table 5.1 Examples of offline and online metrics
|
Offline metrics
|
Online metrics
|
|---|---|
|
Recall at given specificity for spam message classification
|
Number of user complaints about spam messages
|
|
Quantiles of 1.5, 25, 50, 75, 95, and 99
|
Value of expired items, total sales
|
|
Mean reciprocal rank, normalized discounted cumulative gain
|
Click-through rate on search engine result page
|
The table pairs offline measures with outcomes observable in use, including spam complaints, sales, and search-result clicks. section 5.2.2
To assess a change in an online metric, formulate a hypothesis and select key, control, and auxiliary metrics before an A/B test. Report the observed effect, differences across segments, problems encountered, and what a wider rollout might achieve. section 12.5 section 12.0
