A metaparameter is a configuration value that controls an adaptive algorithm rather than the system being optimized. It shapes how the algorithm explores, exploits observed performance, or decides when to stop. section 3.1.2 section 3.4
In adaptive experimentation, metaparameters commonly determine:
- Exploration duration: epsilon-greedy uses c to control how long epsilon remains at or above 1, producing pure exploration. section 3.1.3
- Stopping confidence: Thompson sampling stops when an arm's probability of being best exceeds pstop, such as 0.95. section 3.3
- Operational trade-offs: values set too low can restrict exploration, while values set too high can extend experimentation without benefit. section 3.1.3
Decaying epsilon in an ad selector
A decaying epsilon-greedy implementation sets c = 5, computes an initial epsilon from the business-metric and practical-significance assumptions, and updates epsilon as epsilon = min(1, epsilon_0 / (1 + n)). The algorithm stops when epsilon falls below 0.01; afterward, it accepts the version with the higher mean CTR. Changing the threshold changes running time more than the resulting CTR, while changing c alters the duration of pure exploration. section 3.1.3
Listing 3.7 Epsilon-greedy with decaying epsilon
def epsilon_greedy_decay():
bm_max = 0.01
prac_sig = 0.001
c = 5 #1
epsilon_0 = 2*c*(bm_max/prac_sig)**2 #2
epsilon_stop = 0.01 #2
sum_clicks = 0.0
num_ads = 0.0
sum_a = 0.0
num_a = 0
sum_b = 0.0
num_b = 0
ctr_vs_n = []
epsilons = []
n = 0
selected = None
while True:
epsilon = min(1.0, epsilon_0 / (1.0 + n)) #3
epsilons.append(epsilon)
if epsilon < epsilon_stop: #4
break
select = "Randomize"
if np.random.uniform(0,1) < 1-epsilon:
ctr_a = sum_a/num_a if num_a>0 else 0
ctr_b = sum_b/num_b if num_b>0 else 0
if ctr_a > ctr_b:
select = "A"
selected = "A"
elif ctr_b > ctr_a:
select = "B"
selected = "B"
if select == "Randomize":
if np.random.uniform(0,1) < 0.5:
select = "A"
else:
select = "B"
if select == "A":
clicked = measure_a()
sum_a += clicked
num_a += 1
else:
clicked = measure_b()
sum_b += clicked
num_b += 1
sum_clicks += clicked
num_ads += 1
ctr_vs_n.append(sum_clicks / num_ads)
n += 1
if selected == "B":
accept_reject = "Accept"
else:
accept_reject = "Reject"
return ctr_vs_n, epsilons, accept_reject #5
Inspect the Python implementation to see where c, epsilon_stop, the decay schedule, and the stopping condition enter the control flow. section 3.1.3
