An ad campaign must choose which ad to show before it knows each ad’s true click-through rate. Always choosing the current favorite can lock the campaign into an inferior ad; choosing randomly all the time wastes chances to use what it has learned. Epsilon-greedy mixes the two approaches. section 5.3 section 5.7
On each decision, choose a random action with probability ε (epsilon); otherwise, choose an action with the highest estimated reward. The random choice can also select the current favorite. A larger ε allocates more decisions to exploration, while a smaller ε favors exploitation. section 5.3
A simple bandit implementation repeats three steps:
- Choose an action using the ε-greedy rule. section 5.3
- Observe its reward and increment the action’s selection count. section 5.3
- Update its estimated mean reward: add (reward − current estimate) divided by the action’s count. Earlier observations have more influence than later ones. section 5.3
Choosing an ad
With ε = 0.1, the agent makes a random ad choice on 10% of decisions and otherwise selects the ad with the highest estimated click-through rate. A click gives reward 1 and no click gives 0; after each display, the agent updates the chosen ad’s estimate. section 5.3
Listing 5.4 The Epsilon-Greedy Agent
class EpsilonGreedyAgent:
def __init__(self, num_ads, epsilon=0.1):
self.num_ads = num_ads
self.epsilon = epsilon
self.arm_counts = [0] * num_ads
self.q_values = [0.0] * num_ads
def choose_action(self):
if np.random.rand() < self.epsilon: #A
return np.random.randint(self.num_ads)
else:
return np.argmax(self.q_values) #B
def update(self, chosen_ad_index, reward):
self.arm_counts[chosen_ad_index] += 1
n = self.arm_counts[chosen_ad_index]
current_q = self.q_values[chosen_ad_index]
new_q = current_q + (1/n) * (reward - current_q)
self.q_values[chosen_ad_index] = new_q
The agent implements random versus greedy selection and updates each ad’s estimated reward incrementally. section 5.3
The same idea can provide a non-contextual discounting baseline: estimate the average revenue for each discount across all customers, then explore or exploit those estimates. Unlike a contextual method such as LinUCB, this baseline does not personalize its choice using customer features. section 5.5
Random exploration is simple, but it does not target uncertainty as UCB does. With a fixed positive ε, continued exploration can produce linearly growing cumulative regret. True regret can be checked in a simulation with known reward probabilities, but not calculated exactly in a real campaign where the optimal reward is unknown. section 5.2 section 5.4



