A manufacturer must decide whether a batch meets its advertised fill volume, even though a sample cannot reveal the population mean with certainty. Hypothesis testing turns such evidence into a decision using hypotheses, error rates, and a decision rule specified before analysis. The Neyman–Pearson framework emphasizes repeatable decisions and long-run error control rather than proof. section 4.0 section 4.2.1
A disciplined test moves from design through evaluation to interpretation:
- State the null hypothesis H₀ and alternative H₁. The null often represents the status quo; the alternative states the effect or departure of interest. section 4.1.1
- Choose a significance level α, a suitable statistical test, and a decision rule before analyzing the data. These choices depend on the question, assumptions, data, and consequences of error. section 4.0 section 4.2.4
- Compute the test statistic and compare it with the critical region, or compare its p-value with α. Reject H₀ if the p-value is at or below the threshold; otherwise fail to reject it. section 4.1.6 section 10.2.3
- Interpret the statistical conclusion in context, recognizing that the decision does not remove uncertainty. section 4.5 section 4.2.4
A p-value is the probability, assuming H₀ is true, of a result at least as extreme as the observed result. It is not the probability that H₀ is true. Rejecting H₀ does not establish that H₁ is certainly true, and failing to reject H₀ does not establish that H₀ is true. section 10.2.3 section 4.1.6
A Type I error rejects a true null hypothesis; α controls its long-run probability under the test’s assumptions. A Type II error fails to detect a genuine effect. Statistical power concerns the ability to detect an effect, and depends in part on sample size and effect magnitude. Choosing a threshold requires attention to the practical costs of both errors. section 4.1.6 section 10.2.3 section 4.2.4
Two decisions, different questions
For a 12-ounce beverage target, H₀ states that the mean fill is 12 ounces and H₁ that it differs from 12 ounces. Because deviations in either direction matter, the test is two-tailed. In a small clinical trial, H₀ sets the recovery probability at 0.50 and H₁ specifies a probability greater than 0.50, making the test one-tailed. The trial’s evidence falls just short of its predetermined significance threshold; that does not prove the null hypothesis. section 4.2.1 section 4.2.3
A decision threshold creates a sharp boundary even when evidence changes gradually: p-values of 0.049 and 0.051 are close, yet lead to different decisions at α = 0.05. The rule makes the decision reproducible; it does not eliminate the need for judgment about assumptions and consequences. section 4.1.6 section 4.2.4







