A false negative occurs when a test, classifier, or alert fails to identify a condition or effect that is actually present. In an A/B test, it means overlooking a real difference; in data-quality monitoring, it means failing to alert when data has fallen outside acceptable limits. section 12.2.1 section 14.3.6
Account for false negatives by:
- Choosing a minimum detectable effect no larger than the expected uplift, so the experiment is not designed to overlook the change. section 12.2.1
- Setting the acceptable Type II error rate and using it with variance, significance level, and sample size to plan the test. section 12.2.1
- Testing the planned criteria with A/A and simulated A/B checks before relying on experiment results. section 12.2.1
- Balancing missed problems against unnecessary alerts when defining data-quality constraints and notifications. section 14.3.6
Experiment planning at Supermegaretail
A retail experiment sets the significance level at 5% and the Type II error rate at 10%, then plans its duration and reports simulations checking both error rates. The resulting report includes confidence intervals, methodology, and recommended next steps. section 12.4.1















