The epsilon algorithm is a simple bandit strategy for online model evaluation: track each alternative, select the current best most of the time, and reserve a specified probability for exploration. section 8.2.6
Implement it with an explicit exploration policy:
- Set ε to 0.9, select the best-performing model 90% of the time, and select another model for exploratory testing 10% of the time. section 8.2.6
- Refine the policy by requiring several trials before setting ε to 1, or decay it after each trial, such as ε′ = 0.99 × ε. section 8.2.6
- Use the results to identify the more successful model, but do not treat them as a precise estimate of how much better it is. section 8.2.6
Figure 8.2 Three one-arm bandits with pay-out records, where each block represents one pull. A light record indicates a zero reward, and a dark one indicates a positive reward. Machine A pays frequently, Machine B pays occasionally, and Machine C never pays.
Compare alternatives through payout records: frequent rewards indicate a stronger candidate, occasional rewards indicate uncertainty, and no rewards indicate poor performance. section 8.2.6
