Contextual bandit
A contextual bandit is a sequential decision model in which an agent observes context, selects an action, and receives feedback only about that action.
Experimentation for Engineers
A sequential decision method that selects among alternatives using both observed outcomes and the context associated with each decision.
- How it is used
- Optimizes personalized decisions against short-term business metrics using logged outcomes and online policies.
- How deep it goes
- Works through greedy modeling, online action selection, simulation, epsilon-greedy exploration, and A/B validation; defers full multistep reinforcement learning.
- Examples
- Uses a social-media recommender that predicts viewing time from user features and displayed posts, simulated over 30 days.
- What you can do after
- Build it from scratch as a greedy or epsilon-greedy recommender that logs rewards, refits models, and selects actions.
Read how Experimentation for Engineers explains contextual bandit →
Optimization Algorithms: AI techniques for design, planning, and control problems
A contextual multi-armed bandit is a bandit model in which an agent observes contextual information before choosing an action and receiving action-specific feedback.
- How it is used
- Uses contextual bandits to select rewarded actions from changing information, particularly delivery-truck combinations for a given route.
- How deep it goes
- Works through action generation, probability sampling, training, and testing; the displayed reward function stops short of using route context.
- Examples
- Builds a city-or-interstate truck selector from size, engine, and tire combinations, with noisy rewards and a route-specific test.
- What you can do after
- Apply it via Vowpal Wabbit to generate candidate actions, sample a choice, collect its reward, and test a truck selection.
Machine Learning System Design
A sequential decision method that selects an action using the observed context and learns its outcome or reward.
- How it is used
- Deploys contextual decisions in recommendation systems to select content or advertising that maximizes a reward.
- How deep it goes
- Explains the agent, context, actions, feedback loop, and replay simulation, but does not derive bandit algorithms.
- Examples
- Uses ad-banner selection with user history and CTR, CVR, or eCPM as reward measures.
- What you can do after
- Evaluate recommendation policies offline by constructing a historical replay simulation.
Read how Machine Learning System Design explains contextual bandit →
Related concepts
- multi-armed banditcontrasts with · A multi-armed bandit is a sequential decision problem in which an agent chooses among options with uncertain rewards, balancing reward seeking with gathering information about those options.
- study designmeasured by · Study design is the planned arrangement of procedures, conditions, measurements, and comparisons used to investigate a question and support valid conclusions.
- Vowpal Wabbitused by · Vowpal Wabbit is an open-source machine-learning system supporting scalable online learning and contextual bandit algorithms.
- complexityproduces · Complexity is the degree of difficulty in understanding, analyzing, changing, or operating a system or problem, arising from its structure, behavior, or interacting parts.
- reward signaluses · A reward signal is an evaluative feedback value representing the desirability or payoff of an action, outcome, or state.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.