liveBook · Manning

Uncertainty bonus

As explained in Applied Reinforcement Learning

An additive score component that favors options whose estimated values are less certain or based on fewer observations.

In ad selection, an uncertainty bonus rewards ads whose performance is less known. Upper Confidence Bound (UCB) adds it to an ad’s estimated value: the bonus is larger for ads tried fewer times and shrinks as total trials grow. One implementation uses C × √(log(total trials) / trials for that ad); C controls the weight given to exploration. section 5.3

How Applied Reinforcement Learning covers it

How deep it goes
Defines the bonus through trial counts, logarithmic decay, and a tunable multiplier, and shows its scoring formula.
Examples
Uses ads as options, comparing their estimated click-through values and how often each has been tried.
What you can do after
Apply it by adding a count-based uncertainty bonus to each ad’s estimated value when choosing which ad to show.
How it connects
Connects estimated value and exploration within Upper Confidence Bound (UCB) bandit selection.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.