Policy value is the expected performance of a decision-making policy under the conditions in which it is evaluated. A policy takes context and chooses an action; it might recommend a movie, rank search results, or route a task to a tool. Define value using an outcome that matters, such as click-through rate, watch time, purchase rate, or task success. section 5.4.1
Before exposing users to a candidate policy, off-policy evaluation estimates its value using data collected by another policy. This is not a perfect replay: the logs contain outcomes for actions actually taken, not every alternative action. section 5.4 section 5.11
A practical inverse propensity weighting evaluation follows these steps:
- Define the candidate policy and the outcome to measure. For each logged event, retain its context, selected action, original policy probability, and observed outcome. section 5.4.1
- Run each logged context through the candidate policy and calculate its probability of choosing the action that was actually logged. section 5.4.1
- Weight the observed outcome by the candidate probability divided by the original policy probability, then aggregate weighted outcomes across events. section 5.4.1 section 5.8.1
Movie recommendations
For each historical session, use the logged context and recommended movie to calculate how likely the new policy would have been to recommend that same movie. Weight the observed engagement outcome by the probability ratio and aggregate across sessions to estimate the new policy’s value. section 5.4.1 section 5.8.1
Figure 5.3 illustrates an example of policy value evaluation in a movie recommendation system, illustrating how logged data, such as recommended movies, alternative options, and contextual details, is processed using techniques like inverse propensity weighting to produce actionable metrics such as engagement and retention.
Follow how recommendation logs and contextual details feed a weighted evaluation of engagement and retention. section 5.8.1
Check whether the estimate is trustworthy. Weighting requires accurate logged probabilities and enough overlap between the original and candidate policies. Inspect weight distributions, effective sample size, confidence intervals, and results by segment. Missing context can hide important differences: a search ranking policy might perform well on transactional queries but poorly on informational ones. Capture relevant context while protecting sensitive data. section 5.11 section 5.7.4
