liveBook · Manning

Power analysis

As explained in AI Model Evaluation

Power analysis is a statistical procedure for determining the sample size needed to detect an effect under specified assumptions.

Power analysis determines the sample size required to detect a specified effect in an A/B test. It replaces generic rules of thumb with numbers based on baseline rates and minimum detectable effects. section 6.3.6

Movie recommendation experiment

Before testing recommendation treatments, calculate the required number of eligible users using the primary metrics’ baseline rate and minimum detectable effect. Combine that estimate with user-level randomization, a four-week duration, and exclusions for users in other experiments. Interpret the result alongside guardrails such as watch time, satisfaction, latency, and cost rather than treating a metric change as sufficient evidence of product value. section 6.3.6

How AI Model Evaluation covers it

How deep it goes
Defines the planning inputs and timing but does not explain the statistical calculation or power-analysis formula.
Examples
Applies planning to a movie recommendation test with user-level randomization, eligible users, and a four-week duration.
What it assumes
Assumes defined primary metrics, baseline rates, and minimum detectable effects before calculating the required sample.
What you can do after
Operate/configure an A/B test by replacing generic sample-size rules with an estimate tied to baseline rates and detectable effects.
How it connects
Connects sample size with A/B testing, minimum detectable effect, baseline rate, primary metrics, and guardrail metrics.

Also explained in

Compare how 4 books explain power analysis →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.