A model can improve an offline metric without improving the outcome the business cares about. An A/B test checks its effect in production: a control group keeps the existing system while a treatment group receives the change. Comparing their outcomes helps separate the change’s effect from external changes. Starting with a limited treatment group also reduces the risk of exposing everyone to a broken system. section 12.0 section 12.2
Design the experiment before deploying the change:
- State a hypothesis and choose a goal-linked key metric. Use offline results, benchmarks, or earlier experiments to estimate the expected uplift; also choose control and auxiliary metrics that could reveal unwanted effects. section 12.1 section 12.2.1 section 12.5
- Choose the unit of observation and a splitting strategy. The unit might be a user, session, transaction, or shop; it need not be the same as the unit used to form groups. section 12.2.1 section 12.4.1
- Set the minimum detectable effect no higher than the expected uplift. Choose acceptable false-positive and false-negative rates, then account for metric variation when estimating sample size and duration. Typical thresholds given here are 5% for type I error and 20% for type II error. section 12.2.1
- If needed, rehearse the design. Repeated simulated A/A tests estimate how often the test falsely finds a difference; simulated A/B tests with an added uplift estimate how often it detects one. section 12.2.5
Testing a retail forecasting change
A retailer expects better forecasts to reduce out-of-stock and overstock losses, with an estimated 0.3%–0.7% revenue increase in a pilot group. It forms comparable store groups through distribution centers and uses a single customer check as the statistical observation. Average check is the key proxy for revenue—but only if the number of checks does not change. The plan therefore also tracks checks per day, model updates, offline forecast metrics, daily revenue, and profit, and specifies Welch’s t-test for the comparison. section 12.4.1
Monitor a running experiment for failures rather than waiting for its final metric. Excluding a segment after problems appear is not recommended unless that option was planned; sometimes the safer choice is to stop and investigate. Report the measured effect, segment differences, problems, likely rollout effect, and recommended next steps. Keep results and reporting artifacts reliable: an overwritten set of model results can reverse an apparent conclusion. section 12.3 section 12.5
A conventional A/B test may be infeasible when regulation or logistics prevent dividing customers into groups. Difference-in-differences, synthetic control, interrupted time-series analysis, and regression discontinuity design are named alternatives, not interchangeable procedures; their use requires further investigation. section 12.2.6
