Upper confidence bound (UCB) is a shortcut for choosing actions when calculating full long-term expected utility is too expensive. It favors actions that already look valuable and actions whose value remains uncertain, treating uncertainty as a reason to explore. section 12.10
UCB scores an action by adding its estimated mean utility to an uncertainty bonus: μ is the mean, σ is the standard deviation, and β controls how strongly uncertainty affects the choice. section 12.10
Read the score as a deliberately optimistic estimate:
- A larger μ rewards exploitation: choose actions with higher observed utility. section 12.10
- A larger σ rewards exploration: choose actions whose utility is more uncertain. section 12.10
- A larger β increases the exploration preference; β = 2 can approximate the upper edge of a 95% credible interval for normally distributed utility. section 12.10
Comparing two policies
Suppose policy A produces utilities [4, 9], while policy B usually produces −2 but occasionally produces 28. Policy A has the higher mean, but policy B has much greater uncertainty. With β = 2, UCB selects policy B, matching the decision from Bayesian decision theory.



