Exploitation selects the alternative currently believed to produce the best observed outcome. In a live experiment, it directs most measurements toward the version with the higher measured business metric, such as click-through rate (CTR), so the system benefits from its current best choice while uncertainty remains. section 3.1.2
Exploitation is not the same as certainty. Early measurements may rank versions incorrectly because their standard errors are large. Multi-armed bandit methods therefore pair exploitation with exploration: they capitalize on the apparent winner while continuing to measure alternatives that might actually be better. section 3.1.2 section 3.4
A practical exploitation policy follows this pattern:
- Estimate each alternative’s observed performance from its accumulated measurements. section 3.1.2
- Choose the current leader most of the time, while reserving some probability for random exploration. section 3.1.2
- Reassess as new measurements reduce uncertainty; a previously worse alternative can become the better-supported choice. section 3.1.2
Epsilon-greedy A/B testing
With epsilon = 0.10, epsilon-greedy chooses the version with the higher observed CTR 90% of the time and randomly chooses A or B 10% of the time. In the cited comparison, it achieved about 0.0067 CTR versus about 0.0058 for an A/B test, while repeated runs reached a mean CTR near 0.0068 against B’s 0.0070. section 3.1.2
Listing 3.6 The epsilon-greedy algorithm
def epsilon_greedy(num_ind, epsilon):
sum_clicks = 0.0
num_ads = 0.0
sum_a = 0.0
num_a = 0
sum_b = 0.0
num_b = 0
ctr_vs_n = []
used_b = []
for _ in range(num_ind): #1
select = "Randomize"
if np.random.uniform(0,1) < 1-epsilon: #2
ctr_a = sum_a/num_a if num_a>0 else 0
ctr_b = sum_b/num_b if num_b>0 else 0
if ctr_a > ctr_b: #3
select = "A"
elif ctr_b > ctr_a: #3
select = "B"
# else, if they're equal, randomize
if select == "Randomize": #4
if np.random.uniform(0,1) < 0.5:
select = "A"
else:
select = "B"
if select == "A":
clicked = measure_a()
sum_a += clicked
num_a += 1
used_b.append(False)
else:
clicked = measure_b()
sum_b += clicked
num_b += 1
used_b.append(True)
sum_clicks += clicked
num_ads += 1
ctr_vs_n.append(sum_clicks / num_ads)
return ctr_vs_n, used_b
The implementation makes the decision rule explicit: compare observed CTRs, exploit the larger value, and randomize when exploring or when estimates tie. section 3.1.2
Bayesian optimization applies the same idea to parameter selection. Pure exploitation chooses the parameter minimizing predicted CPU time; a combined heuristic minimizes CPU time minus model uncertainty, selecting a value around 0.11. Continued iterations can stop changing the selected parameter. section 6.1.5
