liveBook · Manning

Backtesting

As explained in AI Model Evaluation

An evaluation technique that compares model predictions with outcomes that actually occurred in historical data.

Backtesting compares a model’s predictions with outcomes that actually occurred in historical data. It supports offline evaluation only when the data reflects real product scenarios and user behavior; otherwise, its metrics can mislead product and engineering decisions. section 1.3.3

Build a useful backtesting setup by separating data roles and checking representativeness:

  • Use historical production events, user behavior, system decisions, labels, outcomes, or interactions as evaluation inputs. section 1.3.3
  • Keep training, validation, and holdout data distinct: training supports learning, validation supports tuning and iteration, and holdout data supports a more independent final assessment. section 1.3.3
  • Treat synthetic data as supplemental: it can cover rare failures or missing cases, but requires validation against realistic production patterns. section 1.3.3

Recommendation evaluation

A recommendation system can be backtested by comparing its predicted recommendations with recorded user interactions. Results become unreliable when the historical data reflects outdated behavior, biased exposure from an earlier model, or incomplete logging. section 1.3.3

How AI Model Evaluation covers it

How deep it goes
Explains historical inputs, dataset roles, and representativeness risks, but does not detail scoring formulas or implementation code.
Examples
Uses production events, user behavior, system decisions, labels, outcomes, interactions, and synthetic rare-failure cases.
What it assumes
Assumes access to historical production records and distinguishes learning, tuning, and independent assessment data.
What you can do after
Evaluate model predictions against historical outcomes while judging whether the data represents realistic product behavior.
How it connects
Connects offline evaluation with historical data, training data, validation data, holdout data, and synthetic data.

Also explained in

Compare how 2 books explain backtesting →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.