Measurements are identically distributed when they can be modeled as draws from the same probability distribution. Together with independence, this forms the iid assumption required for reliable aggregate standard errors and normally distributed z scores in A/B tests. section 8.1.1
Treat the assumption as an operational check:
- Verify that individual measurements share the modeled distribution rather than mixing measurements generated by different processes or conditions. section 8.1.1
- Check independence separately: repeated values caused by sampling too quickly can make the estimated standard error too small and inflate the z score. section 8.1.1
Repeated measurements are a warning sign
An array containing ten copies of 1.5 has standard deviation 0 and estimated standard error 0. An array containing eight copies of 1.5 followed by 2.3 and 3.1 has mean 1.74 and estimated standard error 0.16. These examples expose sampling or measurement problems that can invalidate iid-based analysis. section 8.1.1
