Contextual bandit
A contextual bandit is a sequential decision model in which an agent observes context, selects an action, and receives feedback only about that action.
Experimentation for Engineers
A sequential decision method that selects among alternatives using both observed outcomes and the context associated with each decision.
- How it is used
- Optimizes personalized decisions against short-term business metrics using logged outcomes and online policies.
- How deep it goes
- Works through greedy modeling, online action selection, simulation, epsilon-greedy exploration, and A/B validation; defers full multistep reinforcement learning.
- Examples
- Uses a social-media recommender that predicts viewing time from user features and displayed posts, simulated over 30 days.
- What you can do after
- Build it from scratch as a greedy or epsilon-greedy recommender that logs rewards, refits models, and selects actions.
Read how Experimentation for Engineers explains contextual bandit →
Optimization Algorithms: AI techniques for design, planning, and control problems
A contextual multi-armed bandit is a bandit model in which an agent observes contextual information before choosing an action and receiving action-specific feedback.
- How it is used
- Uses contextual bandits to choose combinations of truck attributes for delivery routes.
- How deep it goes
- Explains the context-action-feedback loop and implements sampling and training, but uses a simulated reward that ignores context.
- Examples
- Uses city and interstate routes with truck size, engine, and tire combinations, plus personalized recommendations and advertising applications.
- What you can do after
- Apply contextual bandits via Vowpal Wabbit to sample and test truck recommendations.
Applied Reinforcement Learning
A contextual bandit is a sequential decision model in which an agent observes static context, selects an action, and receives an immediate reward without action-induced state transitions.
- How it is used
- Frames personalized, immediate-payoff decisions as learning from experience without modeling action-induced state transitions.
- How deep it goes
- Defines context, actions, and rewards, then develops LinUCB alongside a non-contextual baseline and offline evaluation.
- Examples
- Uses personalized discounts, online ad placement, email campaigns, and website layout testing as business examples.
- What you can do after
- Apply LinUCB to personalized discount selection and evaluate a contextual policy offline with logged data.
Read how Applied Reinforcement Learning explains contextual bandit →
Machine Learning System Design
A sequential decision method that selects an action using the observed context and learns its outcome or reward.
- How it is used
- Deploys contextual decisions in recommendation systems to select content or advertising that maximizes a reward.
- How deep it goes
- Explains the agent, context, actions, feedback loop, and replay simulation, but does not derive bandit algorithms.
- Examples
- Uses ad-banner selection with user history and CTR, CVR, or eCPM as reward measures.
- What you can do after
- Evaluate recommendation policies offline by constructing a historical replay simulation.
Read how Machine Learning System Design explains contextual bandit →
Also covered in
What to know first
- multi-armed banditA multi-armed bandit is a sequential decision problem in which an agent repeatedly chooses among alternatives with uncertain rewards, balancing reward seeking with information gathering.
- reinforcement learningReinforcement learning is a machine-learning paradigm in which an agent learns behavior or a decision policy through interaction with an environment and evaluative feedback, such as rewards.
- reward signalA reward signal is a value or feedback indicating the desirability, benefit, or cost associated with an action, outcome, or behavior.
- actionAn action is a choice or operation selected or performed by an actor that can affect a system, environment, or process.
Related concepts
- multi-armed banditcontrasts with · A multi-armed bandit is a sequential decision problem in which an agent repeatedly chooses among alternatives with uncertain rewards, balancing reward seeking with information gathering.
- complexityproduces · Complexity is the degree to which interacting elements or factors make a system or problem difficult to understand, predict, change, operate, or solve.
- study designmeasured by · Study design is the planned arrangement of procedures, conditions, measurements, and comparisons for investigating a question and supporting valid conclusions.
- reward signaluses · A reward signal is a value or feedback indicating the desirability, benefit, or cost associated with an action, outcome, or behavior.
Questions about contextual bandit
What is contextual bandit?
A contextual bandit is a sequential decision model in which an agent observes context, selects an action, and receives feedback only about that action.
Which Manning books explain contextual bandit?
Experimentation for Engineers, Optimization Algorithms: AI techniques for design, planning, and control problems, Applied Reinforcement Learning and Machine Learning System Design.
What should I know before learning contextual bandit?
Manning books that teach contextual bandit usually introduce multi-armed bandit, reinforcement learning, reward signal and action first.
What is the difference between contextual bandit and multi-armed bandit?
A contextual bandit is a sequential decision model in which an agent observes context, selects an action, and receives feedback only about that action. A multi-armed bandit is a sequential decision problem in which an agent repeatedly chooses among alternatives with uncertain rewards, balancing reward seeking with information gathering.
What kinds of contextual bandit are there?
Kinds of contextual bandit include greedy contextual bandit.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.



