liveBook · Manning

Non-stationarity

As explained in Applied Reinforcement Learning

Non-stationarity is a condition in which the statistical relationships governing contexts, actions, or outcomes change over time.

Non-stationarity occurs when the payoffs governing decisions change over time. In a bandit, yesterday’s reliable evidence may lead to poor choices today if customer tastes, campaign performance, or market conditions have shifted. section 5.6

A changing campaign

An ad’s click-through rate can shift as its creative wears out, competitors launch campaigns, or seasonal traffic changes. A bandit that clings to old results may keep choosing the ad even after its payoff falls. section 5.6

Keep learning responsive to change by:

  • Discounting older observations, using a sliding window of recent outcomes, or resetting when change is detected. section 5.6

How Applied Reinforcement Learning covers it

How deep it goes
Defines drift through changing payoffs and names forgetting, sliding windows, and resets without detailing their implementation.
Examples
Uses evolving customer tastes, decaying creative performance, competitors, seasonality, and shifting click-through rates as examples.
What you can do after
Diagnose changing payoffs and choose forgetting, sliding windows, or resets to keep a bandit responsive.
How it connects
Connects non-stationarity to bandit learning, exploration, and covariate shift.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.