liveBook · Manning

Propensity score

As explained in AI Model Evaluation

A propensity score is an estimated probability that a logging policy selects a particular action for a given context.

A propensity score estimates how likely a logging policy was to select a particular action or recommendation for a given context. It helps turn structured interaction logs into inputs for counterfactual evaluation, where observed outcomes are used to assess alternatives that were not selected. section 2.2.1

Use propensity scores carefully when evaluating recommendation or decision-making policies:

  • Log action–outcome pairs, propensity scores, and contextual metadata such as device type, session details, location, or time of day. section 2.2.1
  • Validate that assigned probabilities match the system’s actual decision logic and cover the full range of possible actions. section 5.7.2
  • Watch for probabilities concentrated on one action, because inaccurate weighting can skew alternative-action comparisons and optimization decisions. section 5.7.2

Movie recommendation logs

A recommendation log can record the movie shown, the resulting click or watch, the probability that the logging policy selected that movie, and context such as device or session. Those propensity scores can support inverse propensity scoring or doubly robust estimation, while reminding evaluators that a click may reflect prominent placement rather than genuine preference. section 2.2.1

How AI Model Evaluation covers it

How deep it goes
Defines assigned action probabilities and validation concerns, but stops short of deriving counterfactual estimators.
Examples
Uses movie recommendations, clicks, watches, action–outcome pairs, and contextual metadata as concrete logging examples.
What you can do after
Diagnose counterfactual evaluation by validating probabilities, checking action coverage, and identifying concentrated or biased scores.
How it connects
Connects propensity scores to counterfactual analysis, inverse propensity scoring, doubly robust estimation, and incremental action value.

Also explained in

Compare how 3 books explain propensity score →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.