liveBook · Manning

Hypothesis testing

As explained in Timeless Algorithms, The Foundational Ideas

Hypothesis testing, also called statistical testing, formally evaluates evidence against a null hypothesis under specified assumptions.

A manufacturer must decide whether a batch meets its advertised fill volume, even though a sample cannot reveal the population mean with certainty. Hypothesis testing turns such evidence into a decision using hypotheses, error rates, and a decision rule specified before analysis. The Neyman–Pearson framework emphasizes repeatable decisions and long-run error control rather than proof. section 4.0 section 4.2.1

A disciplined test moves from design through evaluation to interpretation:

  • State the null hypothesis H₀ and alternative H₁. The null often represents the status quo; the alternative states the effect or departure of interest. section 4.1.1
  • Choose a significance level α, a suitable statistical test, and a decision rule before analyzing the data. These choices depend on the question, assumptions, data, and consequences of error. section 4.0 section 4.2.4
  • Compute the test statistic and compare it with the critical region, or compare its p-value with α. Reject H₀ if the p-value is at or below the threshold; otherwise fail to reject it. section 4.1.6 section 10.2.3
  • Interpret the statistical conclusion in context, recognizing that the decision does not remove uncertainty. section 4.5 section 4.2.4

A p-value is the probability, assuming H₀ is true, of a result at least as extreme as the observed result. It is not the probability that H₀ is true. Rejecting H₀ does not establish that H₁ is certainly true, and failing to reject H₀ does not establish that H₀ is true. section 10.2.3 section 4.1.6

A Type I error rejects a true null hypothesis; α controls its long-run probability under the test’s assumptions. A Type II error fails to detect a genuine effect. Statistical power concerns the ability to detect an effect, and depends in part on sample size and effect magnitude. Choosing a threshold requires attention to the practical costs of both errors. section 4.1.6 section 10.2.3 section 4.2.4

Two decisions, different questions

For a 12-ounce beverage target, H₀ states that the mean fill is 12 ounces and H₁ that it differs from 12 ounces. Because deviations in either direction matter, the test is two-tailed. In a small clinical trial, H₀ sets the recovery probability at 0.50 and H₁ specifies a probability greater than 0.50, making the test one-tailed. The trial’s evidence falls just short of its predetermined significance threshold; that does not prove the null hypothesis. section 4.2.1 section 4.2.3

A decision threshold creates a sharp boundary even when evidence changes gradually: p-values of 0.049 and 0.051 are close, yet lead to different decisions at α = 0.05. The rule makes the decision reproducible; it does not eliminate the need for judgment about assumptions and consequences. section 4.1.6 section 4.2.4

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Develops a stepwise testing procedure through hypotheses, decision rules, error risks, and worked applications.
Examples
Uses beverage fill volumes and a small clinical trial to contrast two-tailed and one-tailed hypotheses.
What you can do after
Evaluate and understand evidence against a null hypothesis while interpreting decisions and their error risks.
How it connects
Connects statistical inference, p-values, significance levels, statistical power, and Type I and Type II errors.

Also explained in

Compare how 8 books explain hypothesis testing →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.