liveBook · Manning

Regret

As explained in Applied Reinforcement Learning

Regret is the difference between the reward obtained by an algorithm and the reward it could have obtained by acting optimally over the same decisions.

Regret measures the reward an optimal decision would have earned minus the reward an algorithm actually earned. In ad selection, each suboptimal choice adds missed opportunity; minimizing regret means learning to choose well sooner. section 5.4

Cumulative regret reveals the cost of exploration and premature commitment. Effective policies may accumulate regret while learning, then flatten as they identify better choices; inefficient policies keep accumulating it. Since true regret cannot be calculated in real campaigns, business KPIs serve as proxies. section 5.4 section 5.7

Figure 5.8 Cumulative Regret comparisons of different policies for 500 iterations (for better comparison).

Compare cumulative regret across policies over 500 iterations. section 5.4

How Applied Reinforcement Learning covers it

How deep it goes
Defines per-decision and cumulative regret, describes curve behavior, and notes that real-campaign regret cannot be calculated directly.
Examples
Compares ad-selection policies through cumulative regret curves over 500 iterations.
What you can do after
Evaluate bandit policies by comparing cumulative regret curves; use business KPIs as proxies when true regret cannot be calculated.
How it connects
Connects regret to exploration–exploitation and bandit policies such as UCB and Thompson Sampling.

Also explained in

Compare how 4 books explain regret →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.