A/B testing places a model in a constrained production scenario. A controlled proportion of real cases receives the model’s decisions as population A; untreated or otherwise unprocessed cases form population B. Comparing outcomes and progression estimates whether the model improves real-world results. section 8.2.5
Use A/B testing when live evidence matters, but record the operational and governance trade-offs:
- Treat fresh real-world data and predefined evaluation criteria as the basis for causal evidence, reducing apparent gains from coincidence, optimization, or leakage. section 8.2.5
- Check whether the business process can support intervention, implementation, and measurement before committing to the experiment. section 8.2.5
- Document exposure risks when either the model or the control group may receive poorer treatment, and account for security, policy, legal, and ethical barriers. section 8.2.5
- Evaluate live cases alongside relevant performance measures and nonfunctional requirements rather than relying on accuracy alone. section 8.6
Model versus existing treatment
A small proportion of real cases receives decisions from a new model, while the remaining cases continue without that model. Compare outcomes and progression, then document whether the evidence justifies selection. section 8.2.5
