liveBook · Manning

Contextual bandit

As explained in Applied Reinforcement Learning

A contextual bandit is a sequential decision model in which an agent observes static context, selects an action, and receives an immediate reward without action-induced state transitions.

An online retailer choosing a discount faces a different decision for each customer. A contextual bandit uses information available at that moment to choose an action, observe an immediate reward, and improve future choices. It can learn while operating without a complete model of customer behavior. section 5.5 section 5.1

The context might include customer demographics, device type, or time of day. It is static for the decision: the chosen action does not change it. Context can differ between decisions, however, allowing the bandit to personalize its choices rather than seek one best action overall. section 5.4 section 5.5

A contextual bandit decision has three parts:

  • Context: observable information available before the choice. section 5.5
  • Action: one option from the available set, such as a discount level. section 5.5
  • Reward: the immediate payoff observed after acting, used to update the strategy. section 5.1 section 5.5

The agent balances exploration—trying options to learn how they perform—with exploitation, choosing options that already look promising. Unlike a plain multi-armed bandit, it can make that trade-off differently for different contexts. Unlike a full sequential reinforcement learning problem, it does not model actions changing future states. section 5.4 section 5.1

Figure 5.1 Comparing multi-armed bandits, contextual bandits, and full reinforcement learning.

Compare the role of state and action effects across multi-armed bandits, contextual bandits, and full reinforcement learning. section 5.1

Personalized discounts

For each customer interaction, select a discount from a fixed set using the observed context. The goal is to increase cumulative revenue while learning which offers work for which customers. LinUCB is one approach: it combines an estimated reward with an uncertainty-based exploration term. A non-contextual epsilon-greedy agent can provide a baseline. section 5.5

Before deployment, logged data can support offline, off-policy evaluation. In live campaigns, true regret cannot be calculated, so business metrics serve as proxies. Changing customer behavior and delayed rewards also require attention when maintaining the strategy. section 5.7

How Applied Reinforcement Learning covers it

How deep it goes
Defines context, actions, and rewards, then develops LinUCB alongside a non-contextual baseline and offline evaluation.
Examples
Uses personalized discounts, online ad placement, email campaigns, and website layout testing as business examples.
What you can do after
Apply LinUCB to personalized discount selection and evaluate a contextual policy offline with logged data.
How it connects
Connects multi-armed bandits, reinforcement learning, exploration and exploitation, LinUCB, and off-policy evaluation.

Also explained in

Compare how 4 books explain contextual bandit →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.