liveBook · Manning

Minimum detectable effect

As explained in Machine Learning System Design

The smallest effect size that an experiment is designed to detect with specified statistical power and significance criteria.

Suppose a new search engine is expected to increase clicks. Before running an A/B test, the team needs to decide how small an improvement the test should be able to detect. That threshold is the minimum detectable effect (MDE), also called the minimum detectable change (MDC): the smallest difference between groups that the planned test can reliably detect. Set it no higher than the expected uplift, or the test may miss the improvement it was designed to measure. section 12.2.1 section 12.4.2

Choose the MDE alongside the other inputs to experiment design:

  • Choose a metric and estimate its variance. The metric must quantify progress toward the objective; its variation affects the sample size needed. section 12.2.1
  • Set acceptable error rates. A typical false-positive threshold is α = 0.05; a typical false-negative threshold is β = 0.2, corresponding to 80% power. section 12.2.1
  • Estimate the required sample size per group from the MDE, variance, and error thresholds. Then consider how long it will take to collect those samples, including seasonality. A sample may be a user, session, transaction, or shop, depending on the test. section 12.2.1

Estimating an MDE for search

For a photo-search A/B test, start with results from the existing non-ML search algorithm. Compare offline relevance measures with online click-through and conversion rates, select offline metrics that correlate most strongly with each online metric, and use a simple regression to make an initial MDE estimate. Because this is the first A/B test, that estimate lacks comparative results from earlier variants. Later tests can record offline and online differences between variants to improve it. section 12.4.2

Check the design before launch by simulating it. Repeated A/A tests estimate the false-positive rate. Repeated A/B tests add an uplift—usually the MDE—to one group and measure how often the chosen test detects it. That detection rate estimates power; if simulated error rates miss their targets, revisit the test or sample-size estimate. section 12.2.5

How Machine Learning System Design covers it

How deep it goes
Defines MDE and relates it to error thresholds, variance, sample size, and simulation, without developing a mathematical derivation.
Examples
Uses photo-search click and conversion metrics to estimate an MDE, then simulated A/B tests to check detection rates.
What it assumes
Assumes a testable objective, chosen metric, and expected uplift while defining MDE and the relevant error thresholds.
What you can do after
Practice/perform A/B experiment planning by setting an MDE, estimating sample needs, and checking the design through simulation.
How it connects
Connects MDE to expected uplift, statistical power, type I and type II errors, sample size, and A/B testing.
Cover of Machine Learning System Design
liveBookmanning.com
Read Machine Learning System Design on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machine Learning System Design. The section links point to the parts of the book it draws on.