Selection bias is systematic measurement error caused by collecting samples under different conditions. In A/B comparisons, applying that condition consistently to only one variant creates confounder bias and can reverse the decision. section 2.1.2
Trading costs and time of day
Suppose BYSE is measured before lunch and ASDAQ afterward. Because morning trading adds cost, the experiment can repeatedly report that ASDAQ is cheaper even when BYSE is normally cheaper. Measuring each exchange separately in morning and afternoon exposes the time-of-day effect. section 2.1.2
Reduce selection bias by:
- Randomizing which variant receives each measurement across morning and afternoon, rather than assigning variants to fixed time windows. section 2.1.2
- Taking many measurements to reduce ordinary variation, while recognizing that repeated biased measurements remain consistently wrong. section 2.1.2
