liveBook · Manning

Contextual bandit

A contextual bandit is a sequential decision model in which an agent observes context, selects an action, and receives feedback only about that action.

How 4 Manning books explain it

Experimentation for Engineers

A sequential decision method that selects among alternatives using both observed outcomes and the context associated with each decision.

How it is used
Optimizes personalized decisions against short-term business metrics using logged outcomes and online policies.
How deep it goes
Works through greedy modeling, online action selection, simulation, epsilon-greedy exploration, and A/B validation; defers full multistep reinforcement learning.
Examples
Uses a social-media recommender that predicts viewing time from user features and displayed posts, simulated over 30 days.
What you can do after
Build it from scratch as a greedy or epsilon-greedy recommender that logs rewards, refits models, and selects actions.

Read how Experimentation for Engineers explains contextual bandit →

Optimization Algorithms: AI techniques for design, planning, and control problems

A contextual multi-armed bandit is a bandit model in which an agent observes contextual information before choosing an action and receiving action-specific feedback.

How it is used
Uses contextual bandits to choose combinations of truck attributes for delivery routes.
How deep it goes
Explains the context-action-feedback loop and implements sampling and training, but uses a simulated reward that ignores context.
Examples
Uses city and interstate routes with truck size, engine, and tire combinations, plus personalized recommendations and advertising applications.
What you can do after
Apply contextual bandits via Vowpal Wabbit to sample and test truck recommendations.

Read how Optimization Algorithms: AI techniques for design, planning, and control problems explains contextual bandit →

Applied Reinforcement Learning

A contextual bandit is a sequential decision model in which an agent observes static context, selects an action, and receives an immediate reward without action-induced state transitions.

How it is used
Frames personalized, immediate-payoff decisions as learning from experience without modeling action-induced state transitions.
How deep it goes
Defines context, actions, and rewards, then develops LinUCB alongside a non-contextual baseline and offline evaluation.
Examples
Uses personalized discounts, online ad placement, email campaigns, and website layout testing as business examples.
What you can do after
Apply LinUCB to personalized discount selection and evaluate a contextual policy offline with logged data.

Read how Applied Reinforcement Learning explains contextual bandit →

Machine Learning System Design

A sequential decision method that selects an action using the observed context and learns its outcome or reward.

How it is used
Deploys contextual decisions in recommendation systems to select content or advertising that maximizes a reward.
How deep it goes
Explains the agent, context, actions, feedback loop, and replay simulation, but does not derive bandit algorithms.
Examples
Uses ad-banner selection with user history and CTR, CVR, or eCPM as reward measures.
What you can do after
Evaluate recommendation policies offline by constructing a historical replay simulation.

Read how Machine Learning System Design explains contextual bandit →

Also covered in

What to know first

Related concepts

Questions about contextual bandit

What is contextual bandit?

A contextual bandit is a sequential decision model in which an agent observes context, selects an action, and receives feedback only about that action.

Which Manning books explain contextual bandit?

Experimentation for Engineers, Optimization Algorithms: AI techniques for design, planning, and control problems, Applied Reinforcement Learning and Machine Learning System Design.

What should I know before learning contextual bandit?

Manning books that teach contextual bandit usually introduce multi-armed bandit, reinforcement learning, reward signal and action first.

What is the difference between contextual bandit and multi-armed bandit?

A contextual bandit is a sequential decision model in which an agent observes context, selects an action, and receives feedback only about that action. A multi-armed bandit is a sequential decision problem in which an agent repeatedly chooses among alternatives with uncertain rewards, balancing reward seeking with information gathering.

What kinds of contextual bandit are there?

Kinds of contextual bandit include greedy contextual bandit.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.