Suppose a recommender is meant to increase sales, but it has not been deployed. Gross merchandise value cannot yet be measured as an outcome of the new system. A proxy metric is a measurable stand-in used to estimate progress toward a goal that is harder or slower to observe directly. It is useful only if movement in the proxy reliably predicts movement in the goal. section 5.2.2
Choose a proxy by working backward from the decision it must support:
- Write down the end goal, then break it into stages with metrics for each stage. Use a proxy where a stage cannot be measured directly. section 5.2
- For pre-release comparisons, check whether the offline metric predicts the online metric. A quick, reproducible result is valuable only when its direction of improvement carries through. section 5.2.2
- Test the expected relationship after deployment. Form a hypothesis before the experiment, then measure the business effect rather than treating an improved proxy as the final result. section 12.1 section 12.0
From recommendations to sales
An e-commerce recommender targets higher gross merchandise value. Its proposed path runs through more purchased items and a higher conversion rate. Purchases are sparse: if 3% of offers are clicked and 3% of those clicks become purchases, only about nine of 10,000 offers lead to a purchase. Intermediate measures can provide earlier feedback, but the assumed path to sales still needs online testing. section 5.2.2
A proxy with a condition
In a retail forecast experiment, average check serves as a proxy for revenue only if the number of checks does not change. The experiment therefore also monitors checks per day, alongside revenue and profit. Without that control, a higher average check could give a misleading impression of the total result. section 12.4.1
Table 5.1 Examples of offline and online metrics
|
Offline metrics
|
Online metrics
|
|---|---|
|
Recall at given specificity for spam message classification
|
Number of user complaints about spam messages
|
|
Quantiles of 1.5, 25, 50, 75, 95, and 99
|
Value of expired items, total sales
|
|
Mean reciprocal rank, normalized discounted cumulative gain
|
Click-through rate on search engine result page
|
Compare the offline measures with the online outcomes they are intended to predict, such as spam-classification recall and user complaints. section 5.2.2
