Upper confidence bound
An upper confidence bound is a selection rule that combines an estimated value with an uncertainty bonus to optimistically choose actions or candidates.
Applied Reinforcement Learning
An upper confidence bound method selects actions using an optimistic estimate that combines estimated reward with an uncertainty-based confidence bonus.
- How it is used
- Guides exploration in ad-selection bandits, personalized discounting, and Monte Carlo tree search.
- How deep it goes
- Works through scoring, initialization, parameter choice, and updates, then extends the idea to contextual prediction and tree selection.
- Examples
- Uses simulated ad clicks, revenue from personalized discounts, and vehicle-routing branches to illustrate decisions.
- What you can do after
- Builds UCB and LinUCB agents from scratch for simulated ads and contextual discount decisions.
Read how Applied Reinforcement Learning explains upper confidence bound →
Bayesian Optimization in Action
An upper confidence bound is a Bayesian optimization acquisition policy that combines a predictive mean with an uncertainty-based confidence term to select candidate evaluations.
- How it is used
- Selects the next Bayesian optimization query by optimistically scoring predicted objective values under uncertainty.
- How deep it goes
- Derives μ + βσ from Gaussian-process confidence intervals, explains β behavior, and implements UCB with BoTorch.
- Examples
- Uses weather uncertainty, slot machines, the one-dimensional Forrester function, and support-vector-machine hyperparameter tuning.
- What you can do after
- Apply it via BoTorch by configuring UpperConfidenceBound with a GP model and β, then optimizing its acquisition score.
Read how Bayesian Optimization in Action explains upper confidence bound →
Optimization Algorithms: AI techniques for design, planning, and control problems
Upper confidence bound is an action-selection strategy that combines estimated reward with an uncertainty bonus to balance exploitation and exploration.
- How it is used
- Balances exploration and exploitation when selecting actions in a noncontextual multi-armed bandit.
- How deep it goes
- Implements initialization, a confidence-score formula, iterative updates, parameter tuning, and regret comparison through Python code.
- Examples
- Uses three slot machines, 90 trials, payoff distributions, NumPy listings, and average-regret comparisons.
- What you can do after
- Apply it via NumPy by selecting arms, updating rewards, tuning c, and measuring average regret.
Grokking Bayes
Upper confidence bound is a decision rule that selects actions using an optimistic estimate combining observed value with an uncertainty bonus.
- How it is used
- Approximates costly sequential decision-making by balancing immediate utility with strategic exploration.
- How deep it goes
- Defines the score, explains its components and optimism, and demonstrates a two-policy calculation without deriving full sequential search.
- Examples
- Uses two utility lists, mean and standard-deviation calculations, and a figure where β = 2 selects policy B.
- What you can do after
- Apply UCB by computing each action’s mean and standard deviation, choosing β, and comparing μ + βσ scores.
Also covered in
What to know first
- multi-armed banditA multi-armed bandit is a sequential decision problem in which an agent repeatedly chooses among alternatives with uncertain rewards, balancing reward seeking with information gathering.
- exploitationIn optimization and decision-making, exploitation is selecting an option currently believed, based on available information, to yield high value.
- Bayesian optimizationBayesian optimization is a sequential method for optimizing an expensive or unknown objective by using a probabilistic model to guide evaluations.
- exploration-exploitation trade-offThe exploration-exploitation trade-off is the decision problem of balancing actions that gather information about uncertain alternatives with actions chosen for their currently expected payoff.
Related concepts
- multi-armed bandituses · A multi-armed bandit is a sequential decision problem in which an agent repeatedly chooses among alternatives with uncertain rewards, balancing reward seeking with information gathering.
- acquisition functionuses · An acquisition function is a scoring rule that ranks candidate evaluations by their expected usefulness for selecting the next sample or experiment.
- LinUCBused by · A contextual bandit algorithm that models expected reward as a linear function of context and uses upper confidence bounds to balance exploration and exploitation.
- Monte Carlo Tree Searchused by · Monte Carlo Tree Search is a decision-making algorithm that explores a search tree using repeated sampling or simulation to estimate the value of actions.
- policykind of · A policy is a rule, principle, or decision function that governs behavior or determines actions or permissions under specified conditions.
Questions about upper confidence bound
What is upper confidence bound?
An upper confidence bound is a selection rule that combines an estimated value with an uncertainty bonus to optimistically choose actions or candidates.
Which Manning books explain upper confidence bound?
Applied Reinforcement Learning, Bayesian Optimization in Action, Optimization Algorithms: AI techniques for design, planning, and control problems and Grokking Bayes.
What should I know before learning upper confidence bound?
Manning books that teach upper confidence bound usually introduce multi-armed bandit, exploitation, Bayesian optimization and exploration-exploitation trade-off first.
What is upper confidence bound a kind of?
Upper confidence bound is a kind of policy. A policy is a rule, principle, or decision function that governs behavior or determines actions or permissions under specified conditions.
What is upper confidence bound also called?
Upper confidence bound is also called UCB.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.



