Upper confidence bound

As explained in Bayesian Optimization in Action

An upper confidence bound is a Bayesian optimization acquisition policy that combines a predictive mean with an uncertainty-based confidence term to select candidate evaluations.

Upper confidence bound (UCB) is a Bayesian optimization acquisition policy for choosing the next expensive evaluation. Rather than selecting only the location with the best predicted objective value, it acts optimistically: it selects the candidate whose plausible upper reward is highest. This transfers a decision rule from multi-armed bandits, where an agent chooses among uncertain slot machines, to optimization over a continuous search space. section 5.0 section 5.2

With a Gaussian-process predictive model, the objective value at a candidate has predictive mean μ and standard deviation σ. UCB assigns the acquisition score μ + βσ and evaluates the candidate with the largest score. The mean favors locations already predicted to perform well; the uncertainty term gives poorly known locations an optimistic score, creating an exploration–exploitation trade-off. section 5.2.1 section 5.2.2

Figure 5.6 The UCB of a GP that corresponds to the 95% CI. This bound can be used as the acquisition score of the UCB policy.

The upper edge of the predictive confidence interval becomes the acquisition landscape. Its highest point is the next query, even when that point is uncertain rather than having the highest mean. section 5.2.1

Tune β to control how optimistic the policy is:

• Use a smaller β for more exploitation: the score stays closer to the predictive mean and tends to remain in known high-reward regions. section 5.2.2
• Use a larger β for more exploration: uncertainty contributes more strongly, so the policy is more likely to query away from observed data. section 5.2.2
• Avoid treating β as universally optimal. On the Forrester function, β = 1 became stuck at a local optimum, while β = 2 found the global optimum; excessive exploration can also hurt performance. section 5.2.3
• Consider increasing β as observations accumulate and GP uncertainty tightens, preventing the score from collapsing toward mean-only exploitation. section 5.4.1

Using UCB in a BayesOpt loop

With BoTorch, construct an UpperConfidenceBound policy from a GP model and a positive beta value, then optimize its acquisition score to obtain the next candidate. Replacing another acquisition policy with UCB requires only this policy change in the loop. A hyperparameter-tuning exercise compares fixed β values and an adaptive schedule on a simulated support-vector-machine accuracy surface. section 5.2.3 section 5.4.2

For parallel evaluations, do not simply choose the top sequential UCB scores. Batch BayesOpt reformulates a policy’s utility over the joint multivariate Gaussian predictions for all points in a batch, using the batch’s maximum quantity of interest before averaging. section 7.2.1

How Bayesian Optimization in Action covers it

How deep it goes

Derives μ + βσ from Gaussian-process confidence intervals, explains β behavior, and implements UCB with BoTorch.

Examples

Uses weather uncertainty, slot machines, the one-dimensional Forrester function, and support-vector-machine hyperparameter tuning.

What it assumes

Builds intuition from uncertainty examples while relying on Gaussian-process predictive means, standard deviations, and normal confidence intervals.

What you can do after

Apply it via BoTorch by configuring UpperConfidenceBound with a GP model and β, then optimizing its acquisition score.

How it connects

Connects Bayesian optimization with multi-armed bandits, Gaussian processes, confidence intervals, exploration–exploitation, Probability of Improvement, and Expected Improvement.

liveBookmanning.com

Read Bayesian Optimization in Action on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Optimization Algorithms: AI techniques for design, planning, and control problemsUpper confidence bound is an action-selection strategy that combines estimated reward with an uncertainty bonus to balance exploitation and exploration.

Compare how 2 books explain upper confidence bound →

Related concepts

• acquisition functionuses · An acquisition function is a scoring rule that ranks candidate evaluations by their expected usefulness for selecting the next sample or experiment.
• UCB parameterconfigured by · A UCB parameter is a positive scalar that weights predictive uncertainty in an upper confidence bound score and controls its exploration tendency.
• policykind of · A policy is a rule, strategy, or mapping that governs or determines decisions, actions, permissions, or behavior under specified conditions.

This explainer was generated by Manning's knowledge graph from the text of Bayesian Optimization in Action. The section links point to the parts of the book it draws on.