liveBook · Manning

Multi-armed bandit

Also called MAB

A multi-armed bandit is a sequential decision problem in which an agent repeatedly chooses among alternatives with uncertain rewards, balancing reward seeking with information gathering.

How 6 Manning books explain it

Experimentation for Engineers

A multi-armed bandit is a sequential decision problem of choosing among alternatives with unknown reward distributions while learning which maximizes cumulative reward.

How it is used
Reframes online system-change evaluation as maximizing a business metric while learning which candidate performs best.
How deep it goes
Defines arms and epsilon-greedy, then works through multi-arm selection and simulated CTR outcomes without detailing Thompson sampling.
Examples
Uses an ad selector, click-model arms, simulated clicks, and four alternatives with CTRs from 0.005 to 0.011.
What you can do after
Apply it via epsilon-greedy to allocate production measurements across competing versions while favoring stronger observed metrics.

Read how Experimentation for Engineers explains multi-armed bandit →

Applied Reinforcement Learning

A multi-armed bandit is a sequential decision model in which an agent repeatedly chooses among actions and observes their rewards without modeled state transitions.

How it is used
Frames repeated business choices with immediate payoffs as learning which action performs best without modeling state transitions.
How deep it goes
Defines the decision cycle and trade-off, then implements and compares agents in an ad campaign simulation.
Examples
Uses five banner ads with binary click rewards and hidden click-through rates over 1,000 impressions.
What you can do after
Builds a bandit simulation and selection agents from scratch to experiment with ad selection.

Read how Applied Reinforcement Learning explains multi-armed bandit →

Optimization Algorithms: AI techniques for design, planning, and control problems

A multi-armed bandit is a sequential decision model in which an agent repeatedly chooses among uncertain actions to maximize cumulative reward without changing the underlying state.

How it is used
Introduces bandits as reinforcement learning problems for maximizing reward through repeated choices without state-transition planning.
How deep it goes
Defines the model and exploration trade-off, then compares selection strategies through a simulated regret calculation.
Examples
Simulates 90 choices among three slot machines with specified payoff distributions and compares average regret across strategies.
What you can do after
Evaluate and understand bandit strategies by comparing simulated average regret and adjusting exploration parameters.

Read how Optimization Algorithms: AI techniques for design, planning, and control problems explains multi-armed bandit →

Bayesian Optimization in Action

A multi-armed bandit is a sequential decision problem in which an agent repeatedly selects among actions with uncertain reward distributions to balance information gathering and reward seeking.

How it is used
Frames sequential decision-making under uncertainty and motivates exploration policies for Bayesian optimization.
How deep it goes
Defines actions, unknown reward rates, horizon, and cumulative reward, then contrasts the model with Bayesian optimization.
Examples
Uses casino slot machines, hidden reward probabilities, finite pulls, and an exploration–exploitation comparison.
What you can do after
Apply upper confidence bound or Thompson sampling policies to balance exploration and exploitation.

Read how Bayesian Optimization in Action explains multi-armed bandit →

Delivering Machine Learning Projects: From design to deployment

A multi-armed bandit is an adaptive sequential decision method that allocates trials among alternatives while balancing exploration and reward.

How it is used
Evaluates competing models with live users while limiting exposure to alternatives that appear ineffective.
How deep it goes
Explains reward allocation through a machine analogy and live-feedback examples, without detailing an algorithmic implementation.
Examples
Uses three payout machines and recommended-item purchases as concrete reward examples.
What you can do after
Apply it via live or controlled testing to allocate trials and reduce exposure to weak models.

Read how Delivering Machine Learning Projects: From design to deployment explains multi-armed bandit →

Reinforcement Learning from Human Feedback

A bandit problem is a sequential decision problem in which an action produces an outcome without a multi-step state-transition sequence.

How it is used
Frames RLHF as a single-turn decision problem without state transitions.
How deep it goes
Defines the single-step structure and its reward treatment, without developing bandit algorithms or exploration strategies.
Examples
Uses a sampled prompt, generated completion, response-level reward, and single rollout as the concrete interaction.
What you can do after
Evaluate/understand why RLHF optimization omits the traditional horizon and discount factor.

Read how Reinforcement Learning from Human Feedback explains multi-armed bandit →

Also covered in

What to know first

Related concepts

Questions about multi-armed bandit

What is multi-armed bandit?

A multi-armed bandit is a sequential decision problem in which an agent repeatedly chooses among alternatives with uncertain rewards, balancing reward seeking with information gathering.

Which Manning books explain multi-armed bandit?

Experimentation for Engineers, Applied Reinforcement Learning, Optimization Algorithms: AI techniques for design, planning, and control problems, Bayesian Optimization in Action, Delivering Machine Learning Projects: From design to deployment and Reinforcement Learning from Human Feedback.

What should I know before learning multi-armed bandit?

Manning books that teach multi-armed bandit usually introduce experimentation, AI agent, click-through rate and environment first.

What kinds of multi-armed bandit are there?

Kinds of multi-armed bandit include Thompson sampling.

What is multi-armed bandit a kind of?

Multi-armed bandit is a kind of policy. A policy is a rule, principle, or decision function that governs behavior or determines actions or permissions under specified conditions.

What is multi-armed bandit also called?

Multi-armed bandit is also called MAB.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.