Replay evaluation tests an alternative model or strategy against previously logged interactions in an offline sandbox. It separates experimentation from production traffic, allowing teams to compare versions, inspect behavior, and investigate issues without disrupting users. section 2.1
A replay uses structured historical data rather than new live interactions. Useful records include the action taken, its observed outcome, the probability that the logging policy selected that action, and context such as device, session, location, or time. section 2.2.1
Interpret replay results with these constraints:
- Treat clicks, impressions, watches, purchases, saves, and ratings as observed or proxy outcomes rather than complete measures of preference. section 2.2.1
- Compare model behavior under consistent logged contexts, while recognizing that the original system influenced which actions and outcomes were recorded. section 2.2.1
Movie recommendation replay
A recommendation team reuses logs containing impressions, recommended movies, clicks, watches, and session context to compare a new strategy with the logged strategy before exposing it to users. The resulting metrics indicate behavior on recorded situations, not guaranteed future preference. section 2.2.1 section 2.1
Figure 2.1 visualizes how offline evaluations serve as the central concept, with three key categories—performance evaluations, diagnostic evaluations, and simulation-based evaluations—contributing independently. These categories feed into combined evaluations, which integrate insights from all three to provide a holistic view of model behavior.
Use the offline-evaluation map to place replay alongside performance, diagnostic, simulation-based, and combined evaluations. section 2.1

