A hypothesis is a testable proposition linking an action to a goal, expected metric change, and estimated uplift. For example: deploying a new solution will reduce out-of-stock situations and narrow the gap between predicted and actual sales. section 12.2.1
A useful hypothesis makes the experiment measurable: define the action, problem, metrics, expected uplift, and evidence behind the estimate. Offline evaluation can refine the expected effect before online testing. section 12.2.1 section 12.1
Before running an A/B test, connect the hypothesis to:
- A minimum detectable effect no larger than the expected uplift. section 12.2.1
- Key, control, and auxiliary metrics that distinguish the intended effect from related changes. section 12.5
- Type I and Type II error thresholds, sample size, experiment duration, and seasonality. section 12.2.1
From offline estimate to online test
An offline evaluation suggests that a modified ML solution may improve the target metric. The hypothesis turns that estimate into an A/B-testable claim, after which the observed effect, segment differences, rollout expectation, and next steps can be reported. section 12.1 section 12.5
