liveBook · Manning

Minimum detectable effect

As explained in AI Model Evaluation

The minimum detectable effect is the smallest true change in an outcome that an experiment is designed to detect with specified statistical power and significance.

A minimum detectable effect (MDE) is the smallest true change an experiment is designed to detect with specified statistical power and significance. It turns a product question into a measurable planning target. section 6.3.6

Use the MDE to plan and interpret an online experiment:

  • Choose primary metrics and establish their baseline rates. section 6.3.6
  • Calculate the required sample size with a power analysis rather than a generic rule of thumb. section 6.3.6
  • Compare observed gains with the planned effect while checking guardrails, latency, cost, and satisfaction. section 6.3.6

Movie recommendation A/B test

For a four-week user-level test of contextual recommendation features, calculate sample size from baseline metrics and the MDE for primary outcomes. Treat flat overall results with segment improvement as a follow-up hypothesis, not an automatic launch decision. section 6.3.6

How AI Model Evaluation covers it

How deep it goes
Defines the planning role of MDE and links it to power analysis, but does not derive its statistical formula.
Examples
Uses a movie recommendation test with contextual features, user-level randomization, and product guardrails.
What it assumes
Assumes familiarity with A/B tests, baseline metrics, primary outcomes, and statistical power.
What you can do after
Apply via power analysis to size experiments and judge whether measured product gains justify operational trade-offs.
How it connects
Connects A/B testing to power analysis, baseline rates, primary metrics, and guardrails.

Also explained in

Compare how 2 books explain minimum detectable effect →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.