Epsilon-greedy balances exploration and exploitation by selecting a random action with probability epsilon and the currently highest-valued action otherwise. In a multi-armed bandit, it occasionally tests uncertain slot machines instead of committing permanently to an early, possibly misleading winner. section 12.1.6
Three slot machines
Consider 90 trials with machines whose payoff distributions have means 8, 10, and 5. With epsilon set to 0.1, each trial randomly explores with probability 0.1; otherwise, the implementation selects the machine with the highest estimated mean. The sample run reports average regret of 0.3466, between explore-only regret of 2.246 and exploit-only greedy regret of 0.048. section 12.1.6
A NumPy implementation follows this loop:
- Draw a random number and compare it with epsilon. section 12.1.6
- Select a random arm when exploring; otherwise select the arm with the highest mean payoff. section 12.1.6
- Sample the selected arm’s payoff, calculate regret against the maximum mean reward, and accumulate the result. section 12.1.6
Figure 12.7 Three slot machines as a noncontextual multi-armed bandit (MAB) problem. The payoff distribution’s mean and standard deviations of these three machines are (8,0.5), (10, 0.7), and (5, 0.45), respectively.
Compare the three payoff distributions visually before experimenting with epsilon and cumulative regret. section 12.1.6
A small epsilon preserves discovery while exploitation uses current estimates. The strategy is one option alongside explore-only, exploit-only greedy, and upper confidence bound methods for maximizing cumulative reward. section 12.1.6 section 12.7
