Power analysis estimates the minimum sample size needed to detect an effect of a specified magnitude with a chosen probability at a chosen significance level. An a priori analysis makes this decision before collecting data. section 4.2.2
For a t-test, the calculation commonly combines effect size, statistical power, and alpha. A typical exercise uses a medium effect of 0.5, 80% power, and alpha of .05, then examines how stricter alpha levels change the recommended sample size. section 4.2.2
Use the result as part of a responsible testing plan:
- Check the assumptions of the planned parametric test before interpreting its sample-size recommendation; use validation tests such as Welch’s test when equality of variance is uncertain. section 4.2.1
- Treat the computed value as a minimum and set a goal above it, recognizing that sample-size-sensitive tests can produce unstable or misleading conclusions near a threshold. section 4.2.1
- Report effect sizes such as Cohen’s d alongside statistical significance, and communicate what the findings can and cannot establish to stakeholders. section 4.2.1
- Do not keep collecting data until a desired p-value appears; that practice is identified as p-hacking. section 4.2.1
Comparing Boston and New York weather
A practical workflow imports July 2022 weather data, visualizes the distributions, checks and possibly transforms the t-test assumptions, runs an a priori power analysis with alpha .05, a medium effect of 0.5, and 80% power, and then performs an independent-samples t-test. The findings are summarized for a nontechnical stakeholder. section 4.3.4
from statsmodels.stats.power import TTestIndPower #1
pwr = TTestIndPower() #2
sample = pwr.solve_power(effect_size = 0.5,
power = 0.8,
alpha = 0.05)
print(sample)
Use the statsmodels listing to calculate a two-group t-test sample size from effect size, power, and alpha. section 4.2.2
from statsmodels.stats.power import FTestAnovaPower #1
pwr = FTestAnovaPower()
sample = pwr.solve_power(effect_size = 0.3,
power = 0.8,
alpha = 0.05,
k_groups = 4) #2
print(sample)
Use the statsmodels listing to extend power analysis to an ANOVA with four groups, a small effect of 0.3, 80% power, and alpha .05. section 4.2.1
