Statistical significance evaluates whether an observed difference is unlikely under a null hypothesis. In A/B testing, the null hypothesis says that the group difference is noise rather than a real effect. section 12.2.4
Interpret the result with these checks:
- Compare the p-value with a chosen significance level α, commonly 0.05. A p-value below α indicates statistical significance under the test’s assumptions. section 12.2.4
- Choose the test based on the experiment and data. Welch’s t-test is useful when comparing means with unequal group variances. section 12.2.4
- Do not equate significance with a positive or large outcome. Report the effect, confidence interval, practical metrics, and relevant user or item segments. section 12.3.4
A/B test report
A conversion-rate comparison reports a p-value of 0.0472 and concludes a significant effect, while an average-order-value comparison reports 0.3704 and concludes no significant effect. The decision still requires effect size and confidence intervals. section 12.3.4
Table 12.1 An example of what fields can be included in the reporting table
|
Metric
|
Group A
|
Group B
|
MDE
|
Lift
|
p-value
|
Conclusion
|
|---|---|---|---|---|---|---|
|
CVR
|
75.2%
|
79.8%
|
5%
|
6.12%
|
0.0472
|
+4.2–6.9% (significant)
|
|
AOV
|
$232.2
|
$242.8
|
11%
|
4.57%
|
0.3704
|
No significant effect
|
|
…
|
…
|
…
|
…
|
…
|
…
|
…
|
Browse the reporting table for a compact comparison of group metrics, minimum detectable effect, lift, p-value, confidence range, and conclusion. section 12.3.4
