A propensity score estimates how likely a logging policy was to select a particular action or recommendation for a given context. It helps turn structured interaction logs into inputs for counterfactual evaluation, where observed outcomes are used to assess alternatives that were not selected. section 2.2.1
Use propensity scores carefully when evaluating recommendation or decision-making policies:
- Log action–outcome pairs, propensity scores, and contextual metadata such as device type, session details, location, or time of day. section 2.2.1
- Validate that assigned probabilities match the system’s actual decision logic and cover the full range of possible actions. section 5.7.2
- Watch for probabilities concentrated on one action, because inaccurate weighting can skew alternative-action comparisons and optimization decisions. section 5.7.2
Movie recommendation logs
A recommendation log can record the movie shown, the resulting click or watch, the probability that the logging policy selected that movie, and context such as device or session. Those propensity scores can support inverse propensity scoring or doubly robust estimation, while reminding evaluators that a click may reflect prominent placement rather than genuine preference. section 2.2.1


