Independence means that one random outcome does not affect another. In A/B testing, individual measurements must be independent and identically distributed (iid) for the estimated standard error and z score to be trustworthy. Dependence makes the standard error too small, inflates the z score, and increases the chance of incorrectly accepting version B. section 8.1.1
Repeated or correlated measurements create a practical failure chain:
- Sampling too quickly can record the same server value repeatedly, even when the measurement code appears to collect separate observations. section 8.1.1
- Identical repeated observations produce zero observed standard deviation and therefore an unrealistically precise aggregate estimate. section 8.1.1
- An underestimated standard error enlarges the z score, making threshold-based acceptance of B more likely and increasing false positives. section 8.1.1
A repeated-measurement check
Ten observations all equal to 1.5 have mean 1.5 and standard deviation 0, falsely suggesting perfect precision. Replacing two repeats with 2.3 and 3.1 produces mean 1.74 and standard error about 0.16, revealing greater uncertainty. Thresholded experiments also exhibit regression to the mean: a lucky result can pass initially, while later independent measurements return closer to their expected value. section 8.4.3
