liveBook · Manning

Upper confidence bound

Also called UCB

An upper confidence bound is a selection rule that combines an estimated value with an uncertainty bonus to optimistically choose actions or candidates.

How 4 Manning books explain it

Applied Reinforcement Learning

An upper confidence bound method selects actions using an optimistic estimate that combines estimated reward with an uncertainty-based confidence bonus.

How it is used
Guides exploration in ad-selection bandits, personalized discounting, and Monte Carlo tree search.
How deep it goes
Works through scoring, initialization, parameter choice, and updates, then extends the idea to contextual prediction and tree selection.
Examples
Uses simulated ad clicks, revenue from personalized discounts, and vehicle-routing branches to illustrate decisions.
What you can do after
Builds UCB and LinUCB agents from scratch for simulated ads and contextual discount decisions.

Read how Applied Reinforcement Learning explains upper confidence bound →

Bayesian Optimization in Action

An upper confidence bound is a Bayesian optimization acquisition policy that combines a predictive mean with an uncertainty-based confidence term to select candidate evaluations.

How it is used
Selects the next Bayesian optimization query by optimistically scoring predicted objective values under uncertainty.
How deep it goes
Derives μ + βσ from Gaussian-process confidence intervals, explains β behavior, and implements UCB with BoTorch.
Examples
Uses weather uncertainty, slot machines, the one-dimensional Forrester function, and support-vector-machine hyperparameter tuning.
What you can do after
Apply it via BoTorch by configuring UpperConfidenceBound with a GP model and β, then optimizing its acquisition score.

Read how Bayesian Optimization in Action explains upper confidence bound →

Optimization Algorithms: AI techniques for design, planning, and control problems

Upper confidence bound is an action-selection strategy that combines estimated reward with an uncertainty bonus to balance exploitation and exploration.

How it is used
Balances exploration and exploitation when selecting actions in a noncontextual multi-armed bandit.
How deep it goes
Implements initialization, a confidence-score formula, iterative updates, parameter tuning, and regret comparison through Python code.
Examples
Uses three slot machines, 90 trials, payoff distributions, NumPy listings, and average-regret comparisons.
What you can do after
Apply it via NumPy by selecting arms, updating rewards, tuning c, and measuring average regret.

Read how Optimization Algorithms: AI techniques for design, planning, and control problems explains upper confidence bound →

Grokking Bayes

Upper confidence bound is a decision rule that selects actions using an optimistic estimate combining observed value with an uncertainty bonus.

How it is used
Approximates costly sequential decision-making by balancing immediate utility with strategic exploration.
How deep it goes
Defines the score, explains its components and optimism, and demonstrates a two-policy calculation without deriving full sequential search.
Examples
Uses two utility lists, mean and standard-deviation calculations, and a figure where β = 2 selects policy B.
What you can do after
Apply UCB by computing each action’s mean and standard deviation, choosing β, and comparing μ + βσ scores.

Read how Grokking Bayes explains upper confidence bound →

Also covered in

What to know first

Related concepts

Questions about upper confidence bound

What is upper confidence bound?

An upper confidence bound is a selection rule that combines an estimated value with an uncertainty bonus to optimistically choose actions or candidates.

Which Manning books explain upper confidence bound?

Applied Reinforcement Learning, Bayesian Optimization in Action, Optimization Algorithms: AI techniques for design, planning, and control problems and Grokking Bayes.

What should I know before learning upper confidence bound?

Manning books that teach upper confidence bound usually introduce multi-armed bandit, exploitation, Bayesian optimization and exploration-exploitation trade-off first.

What is upper confidence bound a kind of?

Upper confidence bound is a kind of policy. A policy is a rule, principle, or decision function that governs behavior or determines actions or permissions under specified conditions.

What is upper confidence bound also called?

Upper confidence bound is also called UCB.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.