liveBook · Manning

False negative

As explained in Machine Learning System Design

A classification or alert failing to indicate a condition that is actually present.

A false negative occurs when a test, classifier, or alert fails to identify a condition or effect that is actually present. In an A/B test, it means overlooking a real difference; in data-quality monitoring, it means failing to alert when data has fallen outside acceptable limits. section 12.2.1 section 14.3.6

Account for false negatives by:

  • Choosing a minimum detectable effect no larger than the expected uplift, so the experiment is not designed to overlook the change. section 12.2.1
  • Setting the acceptable Type II error rate and using it with variance, significance level, and sample size to plan the test. section 12.2.1
  • Testing the planned criteria with A/A and simulated A/B checks before relying on experiment results. section 12.2.1
  • Balancing missed problems against unnecessary alerts when defining data-quality constraints and notifications. section 14.3.6

Experiment planning at Supermegaretail

A retail experiment sets the significance level at 5% and the Type II error rate at 10%, then plans its duration and reports simulations checking both error rates. The resulting report includes confidence intervals, methodology, and recommended next steps. section 12.4.1

How Machine Learning System Design covers it

How deep it goes
Explains Type II error through thresholds, sample-size planning, simulations, and alerting tradeoffs without deriving the full statistical formula.
Examples
Uses A/A checks, simulated A/B tests, Supermegaretail experiment planning, and data-quality alerts as concrete examples.
What it assumes
Assumes familiarity with experiments, sample size, metrics, and data-quality constraints while defining Type II error explicitly.
What you can do after
Evaluate experiments and monitoring systems by planning Type II error rates, checking detection power, and reviewing missed-condition risks.
How it connects
Connects false negatives with Type II error, false positives, minimum detectable effect, statistical power, A/B testing, and data quality.

Also explained in

Compare how 16 books explain false negative →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machine Learning System Design. The section links point to the parts of the book it draws on.