A greedy agent chooses the option with the highest estimated payoff. In an ad campaign, it serves the ad with the best estimated click-through rate (CTR), exploiting what it has learned rather than deliberately exploring alternatives. section 5.3
Its decision loop has two parts:
- Try each ad once to establish initial estimates, then choose the ad with the highest estimated value. section 5.3
- Update the chosen ad’s estimate from its observed reward; early successes can make the agent stick with a suboptimal ad. section 5.3
Choosing an ad
If one ad gets several early clicks, its estimated CTR may lead the agent to keep serving it, leaving better-performing alternatives untested. section 5.3
Listing 5.3 The Greedy Agent
class GreedyAgent:
def __init__(self, num_ads):
self.num_ads = num_ads
self.arm_counts = [0] * num_ads
self.q_values = [0.0] * num_ads
def choose_action(self):
for i in range(self.num_ads): #A
if self.arm_counts[i] == 0: #A
return i #A
return np.argmax(self.q_values) #B
def update(self, chosen_ad_index, reward):
self.arm_counts[chosen_ad_index] += 1 #C
n = self.arm_counts[chosen_ad_index] #C
current_q = self.q_values[chosen_ad_index] #C
new_q = current_q + (1/n) *
[CA](reward - current_q) #D
self.q_values[chosen_ad_index] = new_q #D
Inspect the agent’s initialization, action selection, and incremental estimate update. section 5.3
