A counterfactual outcome is what would have happened under a different system decision or state than the one actually observed. In production systems, only the selected action produces a directly logged reward; alternative outcomes remain unknown. section 5.2 section 5.2.1
This missing information creates an online-offline gap. A model trained on historical recommendations learns user responses to the old policy, not necessarily responses to a newly deployed ranking or trading strategy. section 1.3.2
A biased recommendation estimate
Suppose post #1 is shown in three contexts with viewing-time rewards 0.6, 0.9, and 1.3. Linear regression estimates beta_1 = [0.6, 0.9, 1.3]. For User A with context [0,0,1], the predicted reward is 1.3. If post #1 was never shown in that context because post #2 was selected, the fitted coefficient can become beta_1m = [0.6, 0.9, 0], predicting 0.0 instead. The missing alternative observation produces a strongly biased estimate. section 5.2.1
Counterfactual information is managed through deliberate experimentation:
- Use epsilon-greedy exploration to select alternatives occasionally, accepting a short-term reward cost to collect data for better future predictions. section 5.2
- Validate the resulting policy with an A/B test rather than relying only on offline predictions. section 5.0
- Treat simulations as faster and more repeatable than live experiments, but account for bias when real participants respond to deployed actions. section 1.3.3
contexts = [ [1, 0, 0], [0, 1, 0], [0, 0, 1] ] rewards = [ 0.6, 0.9, 1.3 ]
Inspect the toy contexts and rewards used to show how observed action data supports prediction while unobserved alternatives remain unavailable. section 5.2.1
