liveBook · Manning

Historical data

As explained in AI Model Evaluation

Historical interaction data records past user actions and engagement events in a product.

Historical data records past product events, including user interactions, system decisions, labels, and outcomes. It supplies the raw material for offline evaluation: testing a model against pre-collected data without exposing its outputs to live users. section 1.3.3 section 1.3.1

Interaction records can include impressions, clicks, watches, purchases, saves, and ratings. These are useful proxy signals, not perfect measures of preference: a user may click a movie because it was prominently placed, rather than because it was the best recommendation. section 2.2.1

Use historical data with a clear evaluation purpose:

  • Separate training, validation, and holdout data. Reserve the holdout for a more independent assessment after development and tuning. section 1.3.3
  • Check whether the data represents current product behavior. Outdated behavior, biased exposure, and incomplete logging can make even holdout results misleading. section 1.3.3
  • Refresh data and consider temporal splits or out-of-distribution metrics when behavior changes. What counts as recent or out-of-distribution depends on the product. section 2.3.5
  • Treat offline results as a validation step, not proof of live impact. Static data cannot fully capture how users will respond to a new model. section 1.3.1

Evaluating movie recommendations

Use past clicks, watches, saves, or ratings to train a genre-based recommender. Generate ranked lists for users in a holdout dataset and compare them with their observed interactions as proxy relevance signals. Report ranking metrics such as Precision@K and NDCG, and slice results for users with sparse histories or niche preferences. section 2.2.3

Figure 2.4 illustrates the input-output framework for the movie recommendations example. Data, such as user interactions and movie metadata, feeds into the design process, where the evaluation is structured to test the algorithm’s ranking performance. Metrics such as Precision@K and NDCG provide insight into how effectively the algorithm ranks movies, while additional metrics such as diversity or coverage can help reveal whether the recommendations are too narrow or repetitive.

Follow interaction data and movie metadata through the evaluation design to ranking, diversity, and coverage metrics. section 2.2.3

Historical records of system decisions and subsequent outcomes can also support counterfactual evaluation of alternative actions. Estimates depend on data quality, how the original policy collected data, whether it explored enough alternatives, and the similarity between the old and new policies. section 5.2

If offline precision or accuracy is high but live engagement is flat or falling, investigate more than overfitting: stale data, misaligned metrics, or differences between offline and production implementations may also explain the gap. section 2.3.5

How AI Model Evaluation covers it

How deep it goes
Defines interaction signals and dataset splits, then explains representativeness, overfitting, and counterfactual assumptions.
Examples
Uses movie clicks, watches, saves, and ratings to evaluate a genre-based recommender against proxy relevance signals.
What you can do after
Apply it to an offline recommendation evaluation using holdout interactions, ranking metrics, and user slices.
How it connects
Connects historical data to offline evaluation, backtesting, holdout data, temporal splits, and counterfactual evaluation.

Also explained in

Compare how 3 books explain historical data →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.