P-hacking is the misuse of statistical analysis or stopping rules to make an A/B-test result appear significant when the apparent effect is not real. Early stopping is p-hacking in time: repeatedly monitoring a test and stopping when version B looks better than version A. section 8.2
Suppose a team stops a test as soon as z exceeds 1.64, equivalent here to p < .05. A favorable result may then be a false positive. Repeating this across measurements raises the false-positive rate, while a longer test with larger N is appropriate for smaller effects. section 8.2
P-hacking can also occur across metrics:
- Family-wise error is p-hacking in space: examine many business metrics at one point in time and report whichever appears significant. section 8.2
- Early stopping can accept performance-reducing changes, forcing teams to identify suspect changes, back them out, and restart testing. section 8.2
Listing 8.1 Simulate an A/B test and monitor the z score
def z_score_vs_n():
def profit_A(): #1
return np.random.uniform(0,1)
def profit_B(): #1
return np.random.uniform(0,1)
z_scores = []
ind_measurements_A = []
ind_measurements_B = []
for n in range(1, 100):
ind_measurements_A.append(profit_A())
ind_measurements_B.append(profit_B())
a = np.array(ind_measurements_A)
b = np.array(ind_measurements_B)
delta = b.mean() - a.mean() #2
se_delta = np.sqrt(a.std()**2 + b.std()**2) / np.sqrt(n)
z_scores.append(delta / se_delta) #3
return np.array(z_scores)
A simulation recomputes the difference, standard error, and z score after each individual measurement, exposing how monitoring creates opportunities for favorable stopping. section 8.2
