Non-stationarity occurs when the payoffs governing decisions change over time. In a bandit, yesterday’s reliable evidence may lead to poor choices today if customer tastes, campaign performance, or market conditions have shifted. section 5.6
A changing campaign
An ad’s click-through rate can shift as its creative wears out, competitors launch campaigns, or seasonal traffic changes. A bandit that clings to old results may keep choosing the ad even after its payoff falls. section 5.6
Keep learning responsive to change by:
- Discounting older observations, using a sliding window of recent outcomes, or resetting when change is detected. section 5.6
