When choosing an ad, showing the current favorite may earn clicks now, but trying a less familiar ad might reveal a better option. The exploration–exploitation trade-off is the tension between learning about uncertain actions and using what is already known to earn rewards. section 5.2 section 5.3
Exploitation selects the action currently expected to pay off best. Exploration deliberately tries less-understood actions to improve estimates. Too little exploration can lock a decision-maker into a suboptimal choice; too little exploitation wastes opportunities to use what has been learned. The choice matters because actions determine which rewards become available as data. section 5.2
In a multi-armed bandit, actions are tested to learn their rewards while accumulating returns. A contextual bandit also considers information such as a customer profile, so the best action can vary by context. In reinforcement learning with sequential decisions, exploration can reveal how actions affect future states and rewards, not just immediate outcomes. section 5.0 section 5.2
Algorithms balance the two aims in different ways:
- Epsilon-greedy usually chooses the action with the highest estimated reward, but sometimes selects an action at random. section 5.2 section 5.3
- Upper Confidence Bound (UCB) favors actions that are promising or uncertain, using both estimated reward and an exploration term. section 5.2 section 5.3
- Thompson sampling samples from beliefs about each action’s reward and chooses the action with the highest sampled value. section 5.2 section 5.3
Testing five banner ads
An e-commerce campaign has five ads and no historical performance data. Each impression selects one ad; a click earns reward 1 and no click earns 0. The agent learns estimated click-through rates from observed outcomes while trying to maximize clicks over 1,000 impressions. Committing too early risks overlooking a better ad; continuing to test gives the agent more information but may cost clicks in the short term. section 5.3 section 5.2
Listing 5.4 The Epsilon-Greedy Agent
class EpsilonGreedyAgent:
def __init__(self, num_ads, epsilon=0.1):
self.num_ads = num_ads
self.epsilon = epsilon
self.arm_counts = [0] * num_ads
self.q_values = [0.0] * num_ads
def choose_action(self):
if np.random.rand() < self.epsilon: #A
return np.random.randint(self.num_ads)
else:
return np.argmax(self.q_values) #B
def update(self, chosen_ad_index, reward):
self.arm_counts[chosen_ad_index] += 1
n = self.arm_counts[chosen_ad_index]
current_q = self.q_values[chosen_ad_index]
new_q = current_q + (1/n) * (reward - current_q)
self.q_values[chosen_ad_index] = new_q
The epsilon-greedy agent makes the balance concrete: it randomly selects an ad with probability epsilon and otherwise picks the highest estimated value, then updates that estimate from the reward. section 5.3





