A confounder is a condition that affects an outcome and is applied differently and consistently to the system variants being compared. This produces confounder bias: an A/B result that attributes the condition’s effect to the variant. For example, measuring ASDAQ before lunch and BYSE after lunch makes ASDAQ appear cheaper when afternoon trading is actually cheaper. section 2.1.2
Randomized exchange measurement
Randomize the exchange for every trade while measuring both morning and afternoon executions. A coin flip sends each order to ASDAQ or BYSE, balancing time of day across variants without requiring prior knowledge of the bias. The resulting measurements correctly show BYSE as cheaper. section 2.1.2
Control confounders through experimental design and implementation discipline:
- Randomize assignment across the conditions that may affect the outcome; incomplete randomization can leave confounder bias in the measurements. section 8.7 section 2.4
- Keep A and B as similar as possible. Refactoring, an alternate database, or another deployment change can turn the comparison into A versus “B plus an extra change.” section 8.4.1
- Check for confounders even after randomization: randomization balances conditions across variants, but it cannot make an unintended change part of both variants. section 8.4.1
