A rolling dataset is a continuously maintained historical record linking offline evaluation scores with outcomes from online experiments. It tests whether offline improvements reliably predict product results instead of assuming that a metric such as precision@10 predicts CTR. section 7.6.3
Maintain the dataset after every online evaluation:
- Record the model or variant ID and experiment date range. section 7.6.3
- Store offline metrics such as recall@K, precision@K, NDCG, diversity index, fairness gap, and latency. section 7.6.3
- Link those scores to online deltas such as CTR lift, conversion-rate change, retention shift, and engagement depth. section 7.6.3
- Capture traffic allocation, experiment length, product domain, and model maturity, then repeat the process for every experiment. section 7.6.3
- Automate correlation updates so historical predictive power can adjust metric weights in the model-selection rubric. section 7.8.3
From offline score to online outcome
A recommendation variant enters the dataset with its precision@10, recall, diversity, fairness gap, and latency, alongside its experiment period and observed CTR lift, retention shift, and engagement depth. Repeated entries reveal whether improvements in each offline metric consistently accompany better online results. section 7.6.3
Figure 7.2 illustrates the offline to online correlation feedback loop.
Use the feedback loop to connect offline scores, live experiment outcomes, correlation analysis, and subsequent selection decisions. section 7.6.3
