Multi-armed bandit
A multi-armed bandit is a sequential decision problem in which an agent repeatedly chooses among alternatives with uncertain rewards, balancing reward seeking with information gathering.
Experimentation for Engineers
A multi-armed bandit is a sequential decision problem of choosing among alternatives with unknown reward distributions while learning which maximizes cumulative reward.
- How it is used
- Reframes online system-change evaluation as maximizing a business metric while learning which candidate performs best.
- How deep it goes
- Defines arms and epsilon-greedy, then works through multi-arm selection and simulated CTR outcomes without detailing Thompson sampling.
- Examples
- Uses an ad selector, click-model arms, simulated clicks, and four alternatives with CTRs from 0.005 to 0.011.
- What you can do after
- Apply it via epsilon-greedy to allocate production measurements across competing versions while favoring stronger observed metrics.
Read how Experimentation for Engineers explains multi-armed bandit →
Applied Reinforcement Learning
A multi-armed bandit is a sequential decision model in which an agent repeatedly chooses among actions and observes their rewards without modeled state transitions.
- How it is used
- Frames repeated business choices with immediate payoffs as learning which action performs best without modeling state transitions.
- How deep it goes
- Defines the decision cycle and trade-off, then implements and compares agents in an ad campaign simulation.
- Examples
- Uses five banner ads with binary click rewards and hidden click-through rates over 1,000 impressions.
- What you can do after
- Builds a bandit simulation and selection agents from scratch to experiment with ad selection.
Read how Applied Reinforcement Learning explains multi-armed bandit →
Optimization Algorithms: AI techniques for design, planning, and control problems
A multi-armed bandit is a sequential decision model in which an agent repeatedly chooses among uncertain actions to maximize cumulative reward without changing the underlying state.
- How it is used
- Introduces bandits as reinforcement learning problems for maximizing reward through repeated choices without state-transition planning.
- How deep it goes
- Defines the model and exploration trade-off, then compares selection strategies through a simulated regret calculation.
- Examples
- Simulates 90 choices among three slot machines with specified payoff distributions and compares average regret across strategies.
- What you can do after
- Evaluate and understand bandit strategies by comparing simulated average regret and adjusting exploration parameters.
Bayesian Optimization in Action
A multi-armed bandit is a sequential decision problem in which an agent repeatedly selects among actions with uncertain reward distributions to balance information gathering and reward seeking.
- How it is used
- Frames sequential decision-making under uncertainty and motivates exploration policies for Bayesian optimization.
- How deep it goes
- Defines actions, unknown reward rates, horizon, and cumulative reward, then contrasts the model with Bayesian optimization.
- Examples
- Uses casino slot machines, hidden reward probabilities, finite pulls, and an exploration–exploitation comparison.
- What you can do after
- Apply upper confidence bound or Thompson sampling policies to balance exploration and exploitation.
Read how Bayesian Optimization in Action explains multi-armed bandit →
Delivering Machine Learning Projects: From design to deployment
A multi-armed bandit is an adaptive sequential decision method that allocates trials among alternatives while balancing exploration and reward.
- How it is used
- Evaluates competing models with live users while limiting exposure to alternatives that appear ineffective.
- How deep it goes
- Explains reward allocation through a machine analogy and live-feedback examples, without detailing an algorithmic implementation.
- Examples
- Uses three payout machines and recommended-item purchases as concrete reward examples.
- What you can do after
- Apply it via live or controlled testing to allocate trials and reduce exposure to weak models.
Reinforcement Learning from Human Feedback
A bandit problem is a sequential decision problem in which an action produces an outcome without a multi-step state-transition sequence.
- How it is used
- Frames RLHF as a single-turn decision problem without state transitions.
- How deep it goes
- Defines the single-step structure and its reward treatment, without developing bandit algorithms or exploration strategies.
- Examples
- Uses a sampled prompt, generated completion, response-level reward, and single rollout as the concrete interaction.
- What you can do after
- Evaluate/understand why RLHF optimization omits the traditional horizon and discount factor.
Read how Reinforcement Learning from Human Feedback explains multi-armed bandit →
Also covered in
- AI Model EvaluationIn early access (MEAP)
- Timeless Algorithms, The Foundational IdeasIn early access (MEAP)
- Causal Inference for Data Science
- Causal AI
- Designing Deep Learning Systems: A guide for software engineers
What to know first
- experimentationExperimentation is the planned investigation of a question or hypothesis by conducting trials or interventions and observing their outcomes.
- AI agentAn AI agent is a computational system that uses artificial intelligence to interpret information and select or perform actions toward goals while interacting with an environment.
- click-through rateClick-through rate is the proportion of impressions or opportunities to display an item that result in a click on that item.
- environmentAn environment is the external context, system, or conditions within which an agent or application operates and interacts.
Related concepts
- exploitationuses · In optimization and decision-making, exploitation is selecting an option currently believed, based on available information, to yield high value.
- researchuses · A deliberate process of investigating a subject or alternatives to gather information, evaluate evidence, and improve understanding or decisions.
- actionproduces · An action is a choice or operation selected or performed by an actor that can affect a system, environment, or process.
- epsilon-greedyaddressed by · Epsilon-greedy is an action-selection strategy that chooses a random action with a specified probability and otherwise chooses the action with the highest estimated value.
- policykind of · A policy is a rule, principle, or decision function that governs behavior or determines actions or permissions under specified conditions.
- Thompson samplingincludes · Thompson sampling is a Bayesian sequential decision method that samples from posterior beliefs about uncertain rewards or outcomes and selects an action according to the sampled values.
Questions about multi-armed bandit
What is multi-armed bandit?
A multi-armed bandit is a sequential decision problem in which an agent repeatedly chooses among alternatives with uncertain rewards, balancing reward seeking with information gathering.
Which Manning books explain multi-armed bandit?
Experimentation for Engineers, Applied Reinforcement Learning, Optimization Algorithms: AI techniques for design, planning, and control problems, Bayesian Optimization in Action, Delivering Machine Learning Projects: From design to deployment and Reinforcement Learning from Human Feedback.
What should I know before learning multi-armed bandit?
Manning books that teach multi-armed bandit usually introduce experimentation, AI agent, click-through rate and environment first.
What kinds of multi-armed bandit are there?
Kinds of multi-armed bandit include Thompson sampling.
What is multi-armed bandit a kind of?
Multi-armed bandit is a kind of policy. A policy is a rule, principle, or decision function that governs behavior or determines actions or permissions under specified conditions.
What is multi-armed bandit also called?
Multi-armed bandit is also called MAB.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.





