liveBook · Manning

Power analysis

As explained in Experimentation for Engineers

Power analysis is the calculation of the sample size or sensitivity needed to limit the probability of failing to detect a specified effect.

Power analysis sizes an A/B test to detect the smallest practically meaningful effect, reducing the chance of missing an opportunity because the experiment is too small. section 2.3.2

Apply it by:

  • Choose the practical significance threshold, such as a cost difference of 1.0 mip. section 2.3.2

Trading-cost experiment

With practical significance set to 1.0 and equal treatment variability, the basic design returns 7 measurements; adding power analysis returns 16. Real agency experiments may need hundreds or thousands of measurements over one to four weeks. section 2.3.2

How Experimentation for Engineers covers it

How deep it goes
Derives the sample-size inequality and implements it numerically, but does not develop a general power-analysis theory.
Examples
Uses simulated trading systems, execution-cost variability, practical significance of 1.0 mip, and seven-versus-sixteen-measurement designs.
What it assumes
Assumes familiarity with z-scores, standard error, and A/B tests while teaching the practical-significance criterion.
What you can do after
Build it from scratch by estimating variability, selecting a meaningful effect, and calculating a rounded-up measurement count.
How it connects
Connects practical significance, standard error, statistical significance, false-negative rates, and A/B testing.
Cover of Experimentation for Engineers
liveBookmanning.com
Read Experimentation for Engineers on liveBook Get the book at manning.com

Also explained in

Compare how 3 books explain power analysis →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Experimentation for Engineers. The section links point to the parts of the book it draws on.