liveBook · Manning

Exploration-exploitation trade-off

As explained in Applied Reinforcement Learning

The exploration-exploitation trade-off is the tension between gathering information through unfamiliar actions and selecting actions currently expected to yield high rewards.

When choosing an ad, showing the current favorite may earn clicks now, but trying a less familiar ad might reveal a better option. The exploration–exploitation trade-off is the tension between learning about uncertain actions and using what is already known to earn rewards. section 5.2 section 5.3

Exploitation selects the action currently expected to pay off best. Exploration deliberately tries less-understood actions to improve estimates. Too little exploration can lock a decision-maker into a suboptimal choice; too little exploitation wastes opportunities to use what has been learned. The choice matters because actions determine which rewards become available as data. section 5.2

In a multi-armed bandit, actions are tested to learn their rewards while accumulating returns. A contextual bandit also considers information such as a customer profile, so the best action can vary by context. In reinforcement learning with sequential decisions, exploration can reveal how actions affect future states and rewards, not just immediate outcomes. section 5.0 section 5.2

Algorithms balance the two aims in different ways:

  • Epsilon-greedy usually chooses the action with the highest estimated reward, but sometimes selects an action at random. section 5.2 section 5.3
  • Upper Confidence Bound (UCB) favors actions that are promising or uncertain, using both estimated reward and an exploration term. section 5.2 section 5.3
  • Thompson sampling samples from beliefs about each action’s reward and chooses the action with the highest sampled value. section 5.2 section 5.3

Testing five banner ads

An e-commerce campaign has five ads and no historical performance data. Each impression selects one ad; a click earns reward 1 and no click earns 0. The agent learns estimated click-through rates from observed outcomes while trying to maximize clicks over 1,000 impressions. Committing too early risks overlooking a better ad; continuing to test gives the agent more information but may cost clicks in the short term. section 5.3 section 5.2

Listing 5.4 The Epsilon-Greedy Agent
class EpsilonGreedyAgent:
    def __init__(self, num_ads, epsilon=0.1):
        self.num_ads = num_ads
        self.epsilon = epsilon
        self.arm_counts = [0] * num_ads
        self.q_values = [0.0] * num_ads
 
    def choose_action(self):
        if np.random.rand() < self.epsilon: #A
            return np.random.randint(self.num_ads)
        else:
            return np.argmax(self.q_values) #B
 
    def update(self, chosen_ad_index, reward):
        self.arm_counts[chosen_ad_index] += 1
        n = self.arm_counts[chosen_ad_index]
        current_q = self.q_values[chosen_ad_index]
        new_q = current_q + (1/n) * (reward - current_q)
        self.q_values[chosen_ad_index] = new_q

The epsilon-greedy agent makes the balance concrete: it randomly selects an ad with probability epsilon and otherwise picks the highest estimated value, then updates that estimate from the reward. section 5.3

How Applied Reinforcement Learning covers it

How deep it goes
Explains the competing aims and their consequences, then implements and compares bandit strategies in a simulation.
Examples
Uses five banner ads with binary click rewards and unknown click-through rates, alongside a price-testing scenario.
What you can do after
Build bandit agents from scratch to test how exploration and exploitation affect a simulated ad campaign.
How it connects
Connects multi-armed bandits, contextual bandits, reinforcement learning, epsilon-greedy, Upper Confidence Bound, and Thompson sampling.

Also explained in

Compare how 6 books explain exploration-exploitation trade-off →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.