liveBook · Manning

Expected reward

As explained in AI Model Evaluation

Expected reward is the predicted average benefit of a policy over its possible actions and outcomes.

Expected reward estimates the total benefit a new policy could produce, such as revenue or engagement. In off-policy evaluation, it turns logged decisions and outcomes into a counterfactual policy comparison. section 5.4.2

Estimate expected reward by:

  • Defining the policy and the outcome that represents value. section 5.4.2
  • Applying the policy to logged contexts and comparing its action probabilities with those recorded by the original policy. section 5.4.2
  • Using inverse propensity weighting to correct for actions appearing frequently because the original policy preferred them. section 5.4.2

Movie recommendation policy

A quality-focused recommender can be evaluated against a popularity-based logged policy using click rate, watch time, or a combined engagement score. The result includes original policy value, new policy value, and improvement.

How AI Model Evaluation covers it

How deep it goes
Defines policy value and gives a procedural workflow with inverse propensity weighting, but does not develop the estimator beyond its weighting step.
Examples
Uses a movie recommender and evaluates click rate, watch time, and combined engagement score against an original policy.
What it assumes
Requires logged action probabilities, observed outcomes, and contextual metadata describing each decision.
What you can do after
Evaluate a candidate policy offline and compare its estimated value and improvement with the original policy.
How it connects
Connects expected reward to off-policy evaluation, inverse propensity weighting, normalized policy value, click-through rate, and watch time.

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.