liveBook · Manning

Rolling dataset

As explained in AI Model Evaluation

A rolling dataset is a continuously maintained collection of records that incorporates newly available evaluation and operational results over time.

A rolling dataset is a continuously maintained historical record linking offline evaluation scores with outcomes from online experiments. It tests whether offline improvements reliably predict product results instead of assuming that a metric such as precision@10 predicts CTR. section 7.6.3

Maintain the dataset after every online evaluation:

  • Record the model or variant ID and experiment date range. section 7.6.3
  • Store offline metrics such as recall@K, precision@K, NDCG, diversity index, fairness gap, and latency. section 7.6.3
  • Link those scores to online deltas such as CTR lift, conversion-rate change, retention shift, and engagement depth. section 7.6.3
  • Capture traffic allocation, experiment length, product domain, and model maturity, then repeat the process for every experiment. section 7.6.3
  • Automate correlation updates so historical predictive power can adjust metric weights in the model-selection rubric. section 7.8.3

From offline score to online outcome

A recommendation variant enters the dataset with its precision@10, recall, diversity, fairness gap, and latency, alongside its experiment period and observed CTR lift, retention shift, and engagement depth. Repeated entries reveal whether improvements in each offline metric consistently accompany better online results. section 7.6.3

Figure 7.2 illustrates the offline to online correlation feedback loop.

Use the feedback loop to connect offline scores, live experiment outcomes, correlation analysis, and subsequent selection decisions. section 7.6.3

How AI Model Evaluation covers it

How deep it goes
Defines required fields and repeated collection, then supports correlation analysis and automated metric-weight updates.
Examples
Uses recall@K, precision@K, NDCG, diversity, fairness, latency, CTR, conversion, retention, and engagement measurements.
What it assumes
Assumes defined offline and online metrics plus experiment logging that remains consistent across time.
What you can do after
Evaluate/understand whether offline metric improvements reliably predict online product outcomes.
How it connects
Connects offline evaluation, online experimentation, A/B testing, correlation analysis, and model selection.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.