liveBook · Manning

Replay evaluation

As explained in AI Model Evaluation

Replay evaluation is an evaluation technique that reuses previously logged events or interactions to simulate and assess system behavior under an alternative strategy.

Replay evaluation tests an alternative model or strategy against previously logged interactions in an offline sandbox. It separates experimentation from production traffic, allowing teams to compare versions, inspect behavior, and investigate issues without disrupting users. section 2.1

A replay uses structured historical data rather than new live interactions. Useful records include the action taken, its observed outcome, the probability that the logging policy selected that action, and context such as device, session, location, or time. section 2.2.1

Interpret replay results with these constraints:

  • Treat clicks, impressions, watches, purchases, saves, and ratings as observed or proxy outcomes rather than complete measures of preference. section 2.2.1
  • Compare model behavior under consistent logged contexts, while recognizing that the original system influenced which actions and outcomes were recorded. section 2.2.1

Movie recommendation replay

A recommendation team reuses logs containing impressions, recommended movies, clicks, watches, and session context to compare a new strategy with the logged strategy before exposing it to users. The resulting metrics indicate behavior on recorded situations, not guaranteed future preference. section 2.2.1 section 2.1

Figure 2.1 visualizes how offline evaluations serve as the central concept, with three key categories—performance evaluations, diagnostic evaluations, and simulation-based evaluations—contributing independently. These categories feed into combined evaluations, which integrate insights from all three to provide a holistic view of model behavior.

Use the offline-evaluation map to place replay alongside performance, diagnostic, simulation-based, and combined evaluations. section 2.1

How AI Model Evaluation covers it

How deep it goes
Explains batch logs, action-outcome pairs, propensities, and context, but does not specify a particular replay estimator.
Examples
Uses movie recommendation logs containing impressions, clicks, watches, and contextual metadata.
What it assumes
Assumes historical interaction logs and contextual metadata, while establishing offline evaluation as isolated experimentation.
What you can do after
Evaluate model versions offline while interpreting metrics as proxy evidence shaped by the logging policy.
How it connects
Connects replay evaluation to offline evaluation, counterfactual analysis, simulation-based evaluation, and production monitoring.

Also explained in

Compare how 2 books explain replay evaluation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.