The iid assumption requires individual measurements to be independent of one another and drawn from the same probability distribution. A/B-test analysis relies on it when estimating the aggregate standard error, se_delta = sd/√N, and treating z = delta/se_delta as normally distributed. section 8.1.1
Repeated measurements
If a measurement process records the same server value repeatedly, ten observations such as [1.5, 1.5, ..., 1.5] have zero standard deviation and therefore zero estimated standard error. This falsely suggests extreme certainty. Occasional repeated values similarly make the standard error too small, inflate the magnitude of z, and increase false positives. section 8.1.1
Use iid as an operational validity check:
- Inspect collection methods for duplicated observations caused by measurement bugs or sampling faster than the system changes. section 8.1.1
- Verify iid and nonstationarity assumptions, avoid early stopping and cherry-picking, replicate broadly, and validate results with reverse A/B tests and holdouts. section 8.7
ind_meas = np.array([1.5]*10) print (ind_meas) print (ind_meas.mean()) print (ind_meas.std()) [1.5 1.5 1.5 1.5 1.5 1.5 1.5 1.5 1.5 1.5] #1 1.5 0.0 #2
Run the repeated-value array example to observe a mean of 1.5 and a standard deviation of 0.0. section 8.1.1
