A feedback loop forms when a policy’s decisions determine which outcomes enter future training data. In a contextual bandit, biased predictions can cause the policy to avoid actions whose evidence is missing, preserving the bias. section 5.2.1
Avoiding an unseen recommendation
Model 1m predicts zero viewing time when User A receives post #1 because that context-action pair was absent from training data. The policy therefore never shows post #1, never logs the missing outcome, and refitting continues to predict zero. The cycle produces a suboptimal model and policy. section 5.2.1
Figure 5.5 Feedback loop caused by refitting on logged samples. If the logged samples are biased, the fit will produce a biased model. If the model is biased, the policy will make biased decisions. The logs record samples of those biased decisions, and the bias is perpetuated.
Trace the loop from biased logged samples to biased fitting, biased decisions, and perpetuated bias. section 5.2.1
Treat feedback loops as an operational risk:
- Assume logged data may underrepresent feature values, feature combinations, actions, or accurate user rewards. section 5.2.1
- Use exploration to collect samples that counteract policy-driven selection and reduce prediction-model bias. section 5.2.1 section 5.5
- Validate contextual-bandit changes with an A/B test; choose exploration methods while considering tuning and regret trade-offs. section 5.5
