Power analysis determines the sample size required to detect a specified effect in an A/B test. It replaces generic rules of thumb with numbers based on baseline rates and minimum detectable effects. section 6.3.6
Movie recommendation experiment
Before testing recommendation treatments, calculate the required number of eligible users using the primary metrics’ baseline rate and minimum detectable effect. Combine that estimate with user-level randomization, a four-week duration, and exclusions for users in other experiments. Interpret the result alongside guardrails such as watch time, satisfaction, latency, and cost rather than treating a metric change as sufficient evidence of product value. section 6.3.6



