In ad selection, an uncertainty bonus rewards ads whose performance is less known. Upper Confidence Bound (UCB) adds it to an ad’s estimated value: the bonus is larger for ads tried fewer times and shrinks as total trials grow. One implementation uses C × √(log(total trials) / trials for that ad); C controls the weight given to exploration. section 5.3
Uncertainty bonus
An additive score component that favors options whose estimated values are less certain or based on fewer observations.
How Applied Reinforcement Learning covers it
- How deep it goes
- Defines the bonus through trial counts, logarithmic decay, and a tunable multiplier, and shows its scoring formula.
- Examples
- Uses ads as options, comparing their estimated click-through values and how often each has been tried.
- What you can do after
- Apply it by adding a count-based uncertainty bonus to each ad’s estimated value when choosing which ad to show.
- How it connects
- Connects estimated value and exploration within Upper Confidence Bound (UCB) bandit selection.
Related concepts
- upper confidence boundused by · An upper confidence bound is a selection rule that combines an estimated value with an uncertainty bonus to optimistically choose actions or candidates.
This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.
