liveBook · Manning

Policy value

As explained in AI Model Evaluation

Policy value is the expected performance of a decision-making policy if deployed under the evaluation conditions.

Policy value is the expected performance of a decision-making policy under the conditions in which it is evaluated. A policy takes context and chooses an action; it might recommend a movie, rank search results, or route a task to a tool. Define value using an outcome that matters, such as click-through rate, watch time, purchase rate, or task success. section 5.4.1

Before exposing users to a candidate policy, off-policy evaluation estimates its value using data collected by another policy. This is not a perfect replay: the logs contain outcomes for actions actually taken, not every alternative action. section 5.4 section 5.11

A practical inverse propensity weighting evaluation follows these steps:

  • Define the candidate policy and the outcome to measure. For each logged event, retain its context, selected action, original policy probability, and observed outcome. section 5.4.1
  • Run each logged context through the candidate policy and calculate its probability of choosing the action that was actually logged. section 5.4.1
  • Weight the observed outcome by the candidate probability divided by the original policy probability, then aggregate weighted outcomes across events. section 5.4.1 section 5.8.1

Movie recommendations

For each historical session, use the logged context and recommended movie to calculate how likely the new policy would have been to recommend that same movie. Weight the observed engagement outcome by the probability ratio and aggregate across sessions to estimate the new policy’s value. section 5.4.1 section 5.8.1

Figure 5.3 illustrates an example of policy value evaluation in a movie recommendation system, illustrating how logged data, such as recommended movies, alternative options, and contextual details, is processed using techniques like inverse propensity weighting to produce actionable metrics such as engagement and retention.

Follow how recommendation logs and contextual details feed a weighted evaluation of engagement and retention. section 5.8.1

Check whether the estimate is trustworthy. Weighting requires accurate logged probabilities and enough overlap between the original and candidate policies. Inspect weight distributions, effective sample size, confidence intervals, and results by segment. Missing context can hide important differences: a search ranking policy might perform well on transactional queries but poorly on informational ones. Capture relevant context while protecting sensitive data. section 5.11 section 5.7.4

How AI Model Evaluation covers it

How deep it goes
Explains replay and probability-ratio weighting step by step, with overlap and estimate-stability checks.
Examples
Uses movie recommendations and search ranking to illustrate weighted outcomes and context-sensitive performance.
What you can do after
Evaluate and understand a candidate policy’s estimated performance using logged outcomes and reliability checks.
How it connects
Connects policy value to off-policy evaluation, inverse propensity weighting, and incremental action value.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.