Complexity arises when hardware, software, people, networks, physical conditions, and social factors interact, making system behavior difficult to predict from experience alone. With N components, roughly N² pairwise interactions can matter. section 1.3.1
Complexity creates practical engineering risks:
- Contextual bandits add offline modeling complexity and online policy, exploration, ensemble, and safety-constraint complexity. section 5.4
- Multi-step procedures such as response surface methodology and Bayesian optimization create more opportunities for errors at each iteration. section 8.5.1
- Small changes can have uncertain effects, so complex systems require controlled validation rather than relying on reasoning alone. section 5.4 section 8.5.1
Validate a complex experiment
Label the result of a complex procedure as version B and A/B-test it against the previous production version. For stronger evidence, vary users, stocks, infrastructure, or elapsed time; after confirmation, deploy broadly or run a reverse A/B test. section 8.5.1
