liveBook · Manning

Incremental action value

As explained in AI Model Evaluation

The estimated change in outcome attributable to taking a particular alternative action instead of the observed or reference action.

Incremental action value estimates how much better or worse a particular alternative action would perform than the observed or reference action. It supports counterfactual evaluation: instead of asking whether a prediction matched a known label, it asks how another decision policy might have performed. section 5.4

Design the metric around differences between alternatives, not merely factors that predict engagement overall:

  • Identify domain factors that make competing actions meaningfully different. section 5.4.3
  • Compare those factors with historical outcomes at each decision point. section 5.4.3
  • Weight factors according to their ability to distinguish the outcomes of alternatives. section 5.4.3
  • Validate propensity scores because inaccurate action probabilities can skew comparisons, weights, and resulting policy choices. section 5.7.2

Movie recommendations

For a horror fan, genre can create a large incremental gap between a horror and a comedy recommendation. A small rating difference, such as 4.2 versus 4.4 stars, may matter much less. Historical checks can test whether genre, popularity, language, release year, or rating differences explain clicks, watches, or completion; scoring can then amplify meaningful contrasts, such as a large genre-preference boost. section 5.4.3

How AI Model Evaluation covers it

How deep it goes
Explains domain tailoring, historical validation, factor weighting, and propensity-score failure modes, without deriving a formal estimator.
Examples
Uses movie recommendations, genre preference, ratings, clicks, watches, completion, and agentic tool choices as examples.
What you can do after
Evaluate/understand alternative actions by identifying discriminative factors, weighting contrasts, and checking propensity-score reliability.
How it connects
Connects incremental action value to policy value, off-policy evaluation, counterfactual evaluation, and propensity scores.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.