Multi-armed bandit
A multi-armed bandit is a sequential decision problem in which an agent chooses among options with uncertain rewards, balancing reward seeking with gathering information about those options.
Experimentation for Engineers
A multi-armed bandit is a sequential decision problem of choosing among alternatives with unknown reward distributions while learning which maximizes cumulative reward.
- How it is used
- Reframes online system-change evaluation as maximizing a business metric while learning which candidate performs best.
- How deep it goes
- Defines arms and epsilon-greedy, then works through multi-arm selection and simulated CTR outcomes without detailing Thompson sampling.
- Examples
- Uses an ad selector, click-model arms, simulated clicks, and four alternatives with CTRs from 0.005 to 0.011.
- What you can do after
- Apply it via epsilon-greedy to allocate production measurements across competing versions while favoring stronger observed metrics.
Read how Experimentation for Engineers explains multi-armed bandit →
Optimization Algorithms: AI techniques for design, planning, and control problems
A multi-armed bandit is a sequential decision model in which an agent repeatedly chooses among uncertain actions to maximize cumulative reward without changing the underlying state.
- How it is used
- Frames repeated reward-maximizing choices as a reinforcement learning problem without state transitions or long-term planning.
- How deep it goes
- Defines arms and reward distributions, then implements and compares four selection strategies; does not derive their performance guarantees.
- Examples
- Simulates 90 pulls of three slot machines with specified payoff distributions and prints average regret for four strategies.
- What you can do after
- Build it from scratch as a NumPy simulation and compare random, greedy, epsilon-greedy, and UCB choices by average regret.
Bayesian Optimization in Action
A multi-armed bandit is a sequential decision problem in which an agent repeatedly selects among actions with uncertain reward distributions to balance information gathering and reward seeking.
- How it is used
- Frames sequential decision-making under uncertainty and motivates exploration policies for Bayesian optimization.
- How deep it goes
- Defines actions, unknown reward rates, horizon, and cumulative reward, then contrasts the model with Bayesian optimization.
- Examples
- Uses casino slot machines, hidden reward probabilities, finite pulls, and an exploration–exploitation comparison.
- What you can do after
- Apply upper confidence bound or Thompson sampling policies to balance exploration and exploitation.
Read how Bayesian Optimization in Action explains multi-armed bandit →
Delivering Machine Learning Projects: From design to deployment
A multi-armed bandit is an adaptive sequential decision method that allocates trials among alternatives while balancing exploration and reward.
- How it is used
- Evaluates competing models with live users while limiting exposure to alternatives that appear ineffective.
- How deep it goes
- Explains reward allocation through a machine analogy and live-feedback examples, without detailing an algorithmic implementation.
- Examples
- Uses three payout machines and recommended-item purchases as concrete reward examples.
- What you can do after
- Apply it via live or controlled testing to allocate trials and reduce exposure to weak models.
Reinforcement Learning from Human Feedback
A bandit problem is a sequential decision problem in which an action produces an outcome without a multi-step state-transition sequence.
- How it is used
- Frames RLHF as a single-turn decision problem without state transitions.
- How deep it goes
- Defines the single-step structure and its reward treatment, without developing bandit algorithms or exploration strategies.
- Examples
- Uses a sampled prompt, generated completion, response-level reward, and single rollout as the concrete interaction.
- What you can do after
- Evaluate/understand why RLHF optimization omits the traditional horizon and discount factor.
Read how Reinforcement Learning from Human Feedback explains multi-armed bandit →
Related concepts
- exploitationuses · In optimization and decision-making, exploitation is selecting an option currently believed, based on available information, to yield high value.
- epsilon-greedyaddressed by · Epsilon-greedy is an action-selection strategy that chooses a random action with probability epsilon and otherwise selects the currently highest-valued action.
- policykind of · A policy is a rule, strategy, or mapping that governs or determines decisions, actions, permissions, or behavior under specified conditions.
- Thompson samplingincludes · Thompson sampling is a Bayesian sequential decision method that samples from posterior beliefs about uncertain rewards or outcomes and selects an action according to the sampled values.
- actionproduces · An action is a choice or operation selected or performed by an actor that can affect a system or environment or produce an outcome.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.