Epsilon-greedy is a multi-armed bandit allocation rule for evaluating production alternatives while limiting the cost of serving weaker ones. Rather than split traffic equally as in an A/B test, it adapts allocation toward the alternative with the best observed business metric, such as click-through rate (CTR). section 3.0 section 3.1.2
On each decision, exploit with probability 1 − ε by selecting the arm with the highest mean reward observed so far. Explore with probability ε by selecting uniformly at random among the arms. ε is a system-specific metaparameter: ε = 1 behaves like equal randomization, while ε = 0 never deliberately explores. section 3.1.2
Figure 3.6 Using epsilon-greedy to test a click model. With probability 1 - epsilon, epsilon-greedy chooses the click model with the higher CTR so far. Otherwise, with probability epsilon, it randomly chooses between A and B.
The decision flow makes the split explicit: observed CTR determines the usual choice, while ε preserves randomized measurements of both alternatives. section 3.1.2
Testing two click models
An ad selector compares model A, with simulated CTR 0.0050, against model B, with CTR 0.0070. At ε = 0.10, it uses the currently higher-CTR model 90% of the time and randomizes 10% of the time. A simulated run reaches about 0.0067 CTR, above an equal-split test’s roughly 0.0058 while remaining below the known-best model’s 0.0070. section 3.1.1 section 3.1.2
Operate epsilon-greedy with these practical safeguards:
- Track each arm’s reward sum and number of selections; use the resulting mean reward as the exploitation estimate. section 3.1.2
- Keep exploration nonzero long enough to avoid locking onto an early, noisy winner; too little exploration can select the wrong version. section 3.1.3
- For a stopping rule, decay ε approximately as 1/n and stop once it falls below a chosen small threshold; lower thresholds run longer. section 3.1.3
- Extend the same rule to K arms: exploit the highest mean-reward arm or explore uniformly across all available arms. section 3.2
In contextual recommendation, epsilon-greedy also turns a prediction-only policy into an experiment: occasional random recommendations collect outcomes for actions that the greedy policy would not show. Those counterfactual observations can reduce prediction-model bias and feedback loops, at a short-term experimentation cost. section 5.2 section 5.2.2
The method favors business-metric performance during evaluation over the fixed-sample inference goal of a conventional A/B test. It is simple to implement and has optimal asymptotic regret, but tuning ε is system-dependent; Thompson sampling is presented as an alternative with stronger operational metric performance and no such metaparameter. section 3.4
