liveBook · Manning

Hypothesis testing

Also called statistical test, hypothesis test, significance testing, statistical significance test

Hypothesis testing is a formal statistical procedure for evaluating evidence from observed data about a specified hypothesis, often a null hypothesis.

How 8 Manning books explain it

Timeless Algorithms, The Foundational Ideas

Hypothesis testing, also called statistical testing, formally evaluates evidence against a null hypothesis under specified assumptions.

How it is used
Turns statistical evidence into prespecified, repeatable decisions under uncertainty across research and operational settings.
How deep it goes
Develops a stepwise testing procedure through hypotheses, decision rules, error risks, and worked applications.
Examples
Uses beverage fill volumes and a small clinical trial to contrast two-tailed and one-tailed hypotheses.
What you can do after
Evaluate and understand evidence against a null hypothesis while interpreting decisions and their error risks.

Read how Timeless Algorithms, The Foundational Ideas explains hypothesis testing →

Grokking Statistics

A statistical procedure for evaluating evidence against a specified null hypothesis about a population or data-generating process.

How it is used
Tests sample-based claims about population means, proportions, and variances for practical decisions.
How deep it goes
Works through hypotheses, tails, α, p-values, critical regions, and small-sample t-tests without formal derivations.
Examples
Uses soda-can volume, tea identification, and a college exam-score sample to ground testing decisions.
What you can do after
Apply it via Python and SciPy to select tails, calculate p-values, and decide whether to reject H0.

Read how Grokking Statistics explains hypothesis testing →

Effective Data Analysis

Hypothesis testing is a statistical technique for evaluating evidence against a null or other specified hypothesis.

How it is used
Uses statistical tests to evaluate study questions and support stakeholder decisions.
How deep it goes
Introduces test selection, significance, assumptions, and documentation while deferring detailed methods and mathematical logic.
Examples
Uses a gym promotion exercise and a before-and-after Help Center analysis with a repeated measures t-test.
What you can do after
Evaluate and understand test choices, limitations, and how to communicate methods to stakeholders.

Read how Effective Data Analysis explains hypothesis testing →

Causal AI

A hypothesis test is a statistical procedure for evaluating whether observed data provide sufficient evidence against a specified null hypothesis.

How it is used
Tests observed group differences, DAG-implied conditional independences, causal effect estimates, and assumptions behind those estimates.
How deep it goes
Works through null hypotheses, statistics, significance thresholds, and Python tests, while treating rigorous independence-as-alternative testing as mathematically nontrivial.
Examples
Uses transportation-survey chi-squared tests and an online-game comparison of purchases between high- and low-engagement players.
What you can do after
Apply it via pgmpy to challenge DAG independences and via DoWhy to test whether simulated assumption violations refute an estimate.

Read how Causal AI explains hypothesis testing →

Statistics Every Programmer Needs

A hypothesis test is a statistical procedure for evaluating evidence against a specified null hypothesis.

How it is used
Uses statistical tests to assess evidence against a null hypothesis, including checks of variable normality before simple regression.
How deep it goes
Defines p-values and significance decisions, and discusses the conventional threshold and its false-positive and false-negative trade-off.
Examples
Runs Shapiro–Wilk tests on stage1 and stage2 variables and reports each test statistic and p-value.
What you can do after
Evaluate evidence by comparing a test's p-value with a predetermined threshold and deciding whether to reject or fail to reject the null.

Read how Statistics Every Programmer Needs explains hypothesis testing →

Grokking Bayes

A hypothesis test is a statistical procedure that evaluates whether observed data provide sufficient evidence against a specified null hypothesis.

How it is used
Frames observed-data questions as formal comparisons against a null model or between competing hypotheses.
How deep it goes
Explains null hypotheses, test statistics, tail probabilities, significance thresholds, binomial tests, and two-sample t-tests.
Examples
Uses a website redesign, 500 visitors, 63 conversions, a 10% baseline, and a Python binomial-test listing.
What you can do after
Apply a binomial test, compute a p-value, and make a threshold-based decision about evidence against a null hypothesis.

Read how Grokking Bayes explains hypothesis testing →

Machine Learning System Design

A statistical procedure for evaluating whether observed data provide evidence against a specified hypothesis or reference distribution.

How it is used
Uses statistical tests to decide whether A/B results indicate an effect and to detect drift during model monitoring.
How deep it goes
Defines hypotheses, p-values, and significance, then outlines simulation checks; defers detailed guidance on choosing a test.
Examples
Uses unequal-variance group comparisons, repeated simulated experiments, and drift detection as applications.
What you can do after
Evaluate A/B differences against a null hypothesis and validate test sensitivity through simulation.

Read how Machine Learning System Design explains hypothesis testing →

Designing AI Agents

An analytical technique that evaluates candidate explanations or predictions against evidence or observations.

How it is used
Turns a suspected Argus fault into a bounded test plan using supplied sandbox checks.
How deep it goes
Constrains investigation to five iterations and sandbox checks; confirming a fault does not authorize recovery.
Examples
Uses Argus to investigate whether a stated suspicion is a real bug.
What you can do after
Diagnose/debug with it by testing a suspicion through bounded sandbox checks.

Read how Designing AI Agents explains hypothesis testing →

Also covered in

What to know first

Related concepts

Questions about hypothesis testing

What is hypothesis testing?

Hypothesis testing is a formal statistical procedure for evaluating evidence from observed data about a specified hypothesis, often a null hypothesis.

Which Manning books explain hypothesis testing?

Timeless Algorithms, The Foundational Ideas, Grokking Statistics, Effective Data Analysis, Causal AI, Statistics Every Programmer Needs, Grokking Bayes, Machine Learning System Design and Designing AI Agents.

What should I know before learning hypothesis testing?

Manning books that teach hypothesis testing usually introduce categorical data, null hypothesis, p-value and variance first.

What is the difference between hypothesis testing and posterior probability?

Hypothesis testing is a formal statistical procedure for evaluating evidence from observed data about a specified hypothesis, often a null hypothesis. A posterior probability or distribution represents the probability of an event, hypothesis, or unknown quantity conditional on observed evidence and prior information.

What kinds of hypothesis testing are there?

Kinds of hypothesis testing include t-test and chi-square test.

What is hypothesis testing also called?

Hypothesis testing is also called statistical test, hypothesis test, significance testing or statistical significance test.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.