liveBook · Manning

Epsilon-greedy

As explained in Applied Reinforcement Learning

An epsilon-greedy strategy selects a random action with probability epsilon and otherwise selects an action with the highest estimated value.

An ad campaign must choose which ad to show before it knows each ad’s true click-through rate. Always choosing the current favorite can lock the campaign into an inferior ad; choosing randomly all the time wastes chances to use what it has learned. Epsilon-greedy mixes the two approaches. section 5.3 section 5.7

On each decision, choose a random action with probability ε (epsilon); otherwise, choose an action with the highest estimated reward. The random choice can also select the current favorite. A larger ε allocates more decisions to exploration, while a smaller ε favors exploitation. section 5.3

A simple bandit implementation repeats three steps:

  • Choose an action using the ε-greedy rule. section 5.3
  • Observe its reward and increment the action’s selection count. section 5.3
  • Update its estimated mean reward: add (reward − current estimate) divided by the action’s count. Earlier observations have more influence than later ones. section 5.3

Choosing an ad

With ε = 0.1, the agent makes a random ad choice on 10% of decisions and otherwise selects the ad with the highest estimated click-through rate. A click gives reward 1 and no click gives 0; after each display, the agent updates the chosen ad’s estimate. section 5.3

Listing 5.4 The Epsilon-Greedy Agent
class EpsilonGreedyAgent:
    def __init__(self, num_ads, epsilon=0.1):
        self.num_ads = num_ads
        self.epsilon = epsilon
        self.arm_counts = [0] * num_ads
        self.q_values = [0.0] * num_ads
 
    def choose_action(self):
        if np.random.rand() < self.epsilon: #A
            return np.random.randint(self.num_ads)
        else:
            return np.argmax(self.q_values) #B
 
    def update(self, chosen_ad_index, reward):
        self.arm_counts[chosen_ad_index] += 1
        n = self.arm_counts[chosen_ad_index]
        current_q = self.q_values[chosen_ad_index]
        new_q = current_q + (1/n) * (reward - current_q)
        self.q_values[chosen_ad_index] = new_q

The agent implements random versus greedy selection and updates each ad’s estimated reward incrementally. section 5.3

The same idea can provide a non-contextual discounting baseline: estimate the average revenue for each discount across all customers, then explore or exploit those estimates. Unlike a contextual method such as LinUCB, this baseline does not personalize its choice using customer features. section 5.5

Random exploration is simple, but it does not target uncertainty as UCB does. With a fixed positive ε, continued exploration can produce linearly growing cumulative regret. True regret can be checked in a simulation with known reward probabilities, but not calculated exactly in a real campaign where the optimal reward is unknown. section 5.2 section 5.4

How Applied Reinforcement Learning covers it

How deep it goes
Works through action selection and incremental reward updates in an agent implementation, then compares regret and contextual alternatives.
Examples
Uses simulated ad click-through rates and a discount-revenue baseline to demonstrate action selection.
What you can do after
Builds an epsilon-greedy bandit agent from scratch using random action selection and incremental reward estimates.
How it connects
Connects exploration and exploitation to multi-armed bandits, regret, upper confidence bounds, and contextual bandits.

Also explained in

Compare how 4 books explain epsilon-greedy →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.