liveBook · Manning

Chapter 5: Contextual Bandit: Optimizing Stochastic One-step Decisions

What you will learn in chapter 5 of Applied Reinforcement Learning

This introduction motivates bandits as experience-based decision methods and distinguishes pure from contextual bandits.

After this chapter you will be able to

What it builds on

Section by section

Introduction

This introduction motivates bandits as experience-based decision methods and distinguishes pure from contextual bandits.

Key terms: transition probabilities, transition function, dynamic programming, multi-armed bandit, action, reward, beliefs, state transitions, contextual bandit, state

5.1 Bandits as lightweight reinforcement learning

This section explains bandits as lightweight reinforcement learning, compares bandit settings with full reinforcement learning, and illustrates their business applications.

Key terms: bandit algorithms, action, reward, strategy, lightweight reinforcement learning, multi-armed bandit, contextual bandit, state, static context, full reinforcement learning

5.2 Tradeoff between exploitation and exploration

This section explains why learning decision-makers balance exploration and exploitation and outlines how several algorithms approach that balance.

Key terms: exploration, exploitation, long-term cumulative reward, multi-armed bandit, contextual bandit, contexts, future states, ε-greedy, Upper Confidence Bound, UCB

5.3 Simulating an Ad campaign problem with bandit algorithms

This section simulates an ad campaign and compares bandit strategies for learning which ad earns the most clicks.

Key terms: arms, action, reward, click-through rate, environment, random agent, greedy agent, ε-greedy, exploration, exploitation

5.4 Quantifying bandit algorithms performance with Regret

This section explains regret as a measure of bandit performance, its relationship to exploration and exploitation, and its practical use in comparing algorithms.

Key terms: regret, reward, optimal reward, exploration, exploitation, cumulative regret, UCB, Thompson Sampling, true CTRs, theoretical regret

5.5 Dynamic personalized discounting with contextual bandits

This section models personalized discounting with LinUCB and compares it with a non-contextual baseline using historical customer data.

Key terms: contextual bandit, context, state, action, reward, LinUCB, context features, expected reward, weights, evidence matrix

5.6 Beyond stationary bandit problems

This section explains practical challenges and design principles for keeping bandit systems useful beyond stationary, immediate-feedback settings.

Key terms: stationary, drift, non-stationarity, forgetting mechanisms, sliding windows, delayed rewards, proxy rewards, pending outcomes, multiple objectives, shape rewards

5.7 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 4: Perfect Knowledge, Optimal Policy: Dynamic Programming Chapter 6: Tabular Reinforcement Learning →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.