liveBook · Manning

Greedy agent

As explained in Applied Reinforcement Learning

A greedy agent always selects the action with the highest current estimated payoff and does not deliberately explore alternatives.

A greedy agent chooses the option with the highest estimated payoff. In an ad campaign, it serves the ad with the best estimated click-through rate (CTR), exploiting what it has learned rather than deliberately exploring alternatives. section 5.3

Its decision loop has two parts:

  • Try each ad once to establish initial estimates, then choose the ad with the highest estimated value. section 5.3
  • Update the chosen ad’s estimate from its observed reward; early successes can make the agent stick with a suboptimal ad. section 5.3

Choosing an ad

If one ad gets several early clicks, its estimated CTR may lead the agent to keep serving it, leaving better-performing alternatives untested. section 5.3

Listing 5.3 The Greedy Agent
class GreedyAgent:
    def __init__(self, num_ads):
        self.num_ads = num_ads
        self.arm_counts = [0] * num_ads
        self.q_values = [0.0] * num_ads
 
    def choose_action(self):
        for i in range(self.num_ads): #A
            if self.arm_counts[i] == 0: #A
                return i #A
        return np.argmax(self.q_values) #B
 
    def update(self, chosen_ad_index, reward):
        self.arm_counts[chosen_ad_index] += 1 #C
        n = self.arm_counts[chosen_ad_index] #C
        current_q = self.q_values[chosen_ad_index] #C
        new_q = current_q + (1/n) *
                [CA](reward - current_q) #D
        self.q_values[chosen_ad_index] = new_q #D

Inspect the agent’s initialization, action selection, and incremental estimate update. section 5.3

How Applied Reinforcement Learning covers it

How deep it goes
Defines the choice rule and demonstrates initialization and incremental updates, while highlighting the risk of locking onto a suboptimal ad.
Examples
Uses ad selection, estimated click-through rates, and early clicks to illustrate action choice and premature commitment.
What you can do after
Build it from scratch by tracking action counts and estimated values, selecting the current maximum, and updating its estimate after rewards.
How it connects
Contrasts exploitation with exploration and compares greedy choice with epsilon-greedy, upper confidence bound, and Thompson sampling.

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.