liveBook · Manning

Chapter 4: From evidence to decisions: Neyman, Pearson, and the principles of hypothesis testing

What you will learn in chapter 4 of Timeless Algorithms, The Foundational Ideas

The introduction explains why learning from data is not enough to make decisions and how the Neyman–Pearson framework adds explicit, advance rules for controlling errors.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The introduction explains why learning from data is not enough to make decisions and how the Neyman–Pearson framework adds explicit, advance rules for controlling errors.

Key terms: inference, false positives, false negatives, Neyman-Pearson framework, hypotheses, error rates, decision rules

4.1 The stepwise framework of hypothesis testing

This section frames Neyman and Pearson’s mathematical approach as a practical, interrelated workflow for moving from a research question to a statistically supported conclusion.

Key terms: hypothesis testing, seven interrelated steps, significance level

4.1.1 Step 1: state the hypotheses

This section explains how to define competing hypotheses and fix the test’s structure before analyzing data, so evidence can support an objective decision.

Key terms: null hypothesis, alternative hypothesis, status quo, simple, composite, parameter of interest, one-sided, two-sided, hypotheses, structure of the test

4.1.2 Step 2: choose a significance level

This section explains how to choose a significance level before testing, based on the consequences of false positives and false negatives.

Key terms: significance level, α, Type I error, Type II error, null hypothesis, false positive, false positives, false negatives, 0.05, conservative

4.1.3 Step 3: select the test

This section explains how to choose a statistical test that fits the question and data, commit to it before analysis, and prefer greater power when valid alternatives exist.

Key terms: research question, type of data, assumptions, statistical power, design decision, decision rule, predetermined rules, Neyman–Pearson Lemma, power, Type I error

4.1.4 Step 4: compute the test statistic

This section explains how the selected test turns observed data into a test statistic and why that value is not yet a decision about the hypotheses.

Key terms: test statistic, observed data, null hypothesis, statistical test

4.1.5 Step 5: establish the decision rule

This section explains how a predetermined significance level becomes a decision rule for evaluating a hypothesis-test statistic.

Key terms: significance level, α, Type I error, critical region, sampling distribution, two-tailed test, critical values, p-value, null hypothesis, decision rule

4.1.6 Step 6: make a decision

This section explains how a hypothesis test reaches a predefined reject-or-fail-to-reject decision while preserving uncertainty about the truth and practical meaning of the result.

Key terms: test statistic, p-value, null hypothesis, reject, fail to reject, decision rule, Type I error, Type II error, significance level, α

4.1.7 Step 7: draw a conclusion

This section explains how to turn a hypothesis-test decision into a contextual conclusion that communicates evidence, uncertainty, and practical importance.

Key terms: null hypothesis, alternative hypothesis, fail to reject, statistical power, research question, limitations, assumptions, significance level, statistical test, statistical significance

4.1.8 From framework to practice

This section explains why Neyman and Pearson’s seven-step framework makes hypothesis testing a disciplined and reproducible way to draw conclusions under uncertainty.

Key terms: seven-step framework, hypothesis testing, risk of error

4.2 Hypothesis testing in action

Introduces how the same seven-step hypothesis-testing framework applies across different problems to support consistent, reproducible decisions.

Key terms: seven-step framework, Neyman–Pearson procedure, reproducible decision-making

4.2.1 Example 1: beverage fill volumes

The section works through a beverage fill-volume test to show how preselected hypotheses and error limits turn sample evidence into a decision without claiming proof.

Key terms: null hypothesis, alternative hypothesis, two-tailed hypothesis test, simple hypothesis, composite hypothesis, significance level, α, Type I error, one-sample, two-tailed t-test, population standard deviation

4.2.2 Example 2: safety monitoring

This section applies the seven-step hypothesis-testing framework to an aircraft vibration reading, showing how a safety-conscious test turns unusual evidence into a precautionary inspection decision.

Key terms: null hypothesis, alternative hypothesis, one-tailed hypothesis test, simple, composite, significance level, α = 0.10, Type I error, Type II error, false positives

4.2.3 Example 3: a small clinical trial

A small clinical trial shows how a prespecified hypothesis test yields a reproducible decision when the evidence narrowly misses statistical significance, without resolving whether the drug works.

Key terms: null hypothesis, alternative hypothesis, one-tailed exact binomial test, binary outcomes, significance level, rejection region, test statistic, p-value, fail to reject, p-hacking

4.2.4 From examples to principles

The section draws three principles from the examples: hypothesis testing uses a consistent, pre-specified decision framework, adapts its choices to context, and supports decisions without proving hypotheses.

Key terms: seven-step procedure, hypotheses, significance levels, statistical tests, decision rules, Type I versus Type II errors, significance level, underlying assumptions, subject-matter knowledge, statistical evidence

4.3 Why it matters

This section explains why the Neyman–Pearson framework matters as a way to turn statistical evidence into decisions while acknowledging the possibility of error.

Key terms: Neyman–Pearson framework, statistical evidence, decisions under uncertainty, possibility of error

4.3.1 From evidence to action

This section explains how the Neyman–Pearson framework turns statistical evidence into a basis for decisions under uncertainty.

Key terms: Neyman–Pearson framework, predefined decision rules, Type I and Type II errors, statistical evidence

4.3.2 Making decisions reproducible

This section explains how pre-specified hypothesis-testing rules make statistical decisions reproducible and protect against conclusions shaped by flexible analysis.

Key terms: Neyman–Pearson framework, hypotheses, significance level, statistical test, decision rule, reproducible, p-hacking, statistical significance, statistical inference

4.3.3 A framework beyond statistics

The section explains why Neyman and Pearson’s decision-making framework remains useful even as statistical methods and fields change.

Key terms: Neyman–Pearson framework, significance level, pre-specified decision rule

4.4 An enduring framework for decision-making

The section explains why the Neyman–Pearson framework remains useful for turning uncertain evidence into disciplined decisions across changing fields and technologies.

Key terms: Neyman–Pearson framework, pre-specified decision rule, statistical conclusion, routine process variation, random variation, model drift, anomaly detection

4.5 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 3: The algorithm of estimation: Ronald Fisher and maximum likelihood Chapter 5: Entropy and the bit: Claude Shannon and the quantification of information →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.