An online retailer choosing a discount faces a different decision for each customer. A contextual bandit uses information available at that moment to choose an action, observe an immediate reward, and improve future choices. It can learn while operating without a complete model of customer behavior. section 5.5 section 5.1
The context might include customer demographics, device type, or time of day. It is static for the decision: the chosen action does not change it. Context can differ between decisions, however, allowing the bandit to personalize its choices rather than seek one best action overall. section 5.4 section 5.5
A contextual bandit decision has three parts:
- Context: observable information available before the choice. section 5.5
- Action: one option from the available set, such as a discount level. section 5.5
- Reward: the immediate payoff observed after acting, used to update the strategy. section 5.1 section 5.5
The agent balances exploration—trying options to learn how they perform—with exploitation, choosing options that already look promising. Unlike a plain multi-armed bandit, it can make that trade-off differently for different contexts. Unlike a full sequential reinforcement learning problem, it does not model actions changing future states. section 5.4 section 5.1
Figure 5.1 Comparing multi-armed bandits, contextual bandits, and full reinforcement learning.
Compare the role of state and action effects across multi-armed bandits, contextual bandits, and full reinforcement learning. section 5.1
Personalized discounts
For each customer interaction, select a discount from a fixed set using the observed context. The goal is to increase cumulative revenue while learning which offers work for which customers. LinUCB is one approach: it combines an estimated reward with an uncertainty-based exploration term. A non-contextual epsilon-greedy agent can provide a baseline. section 5.5
Before deployment, logged data can support offline, off-policy evaluation. In live campaigns, true regret cannot be calculated, so business metrics serve as proxies. Changing customer behavior and delayed rewards also require attention when maintaining the strategy. section 5.7



