Hypothesis testing
Hypothesis testing is a formal statistical procedure for evaluating evidence from observed data about a specified hypothesis, often a null hypothesis.
Timeless Algorithms, The Foundational Ideas
Hypothesis testing, also called statistical testing, formally evaluates evidence against a null hypothesis under specified assumptions.
- How it is used
- Turns statistical evidence into prespecified, repeatable decisions under uncertainty across research and operational settings.
- How deep it goes
- Develops a stepwise testing procedure through hypotheses, decision rules, error risks, and worked applications.
- Examples
- Uses beverage fill volumes and a small clinical trial to contrast two-tailed and one-tailed hypotheses.
- What you can do after
- Evaluate and understand evidence against a null hypothesis while interpreting decisions and their error risks.
Read how Timeless Algorithms, The Foundational Ideas explains hypothesis testing →
Grokking Statistics
A statistical procedure for evaluating evidence against a specified null hypothesis about a population or data-generating process.
- How it is used
- Tests sample-based claims about population means, proportions, and variances for practical decisions.
- How deep it goes
- Works through hypotheses, tails, α, p-values, critical regions, and small-sample t-tests without formal derivations.
- Examples
- Uses soda-can volume, tea identification, and a college exam-score sample to ground testing decisions.
- What you can do after
- Apply it via Python and SciPy to select tails, calculate p-values, and decide whether to reject H0.
Effective Data Analysis
Hypothesis testing is a statistical technique for evaluating evidence against a null or other specified hypothesis.
- How it is used
- Uses statistical tests to evaluate study questions and support stakeholder decisions.
- How deep it goes
- Introduces test selection, significance, assumptions, and documentation while deferring detailed methods and mathematical logic.
- Examples
- Uses a gym promotion exercise and a before-and-after Help Center analysis with a repeated measures t-test.
- What you can do after
- Evaluate and understand test choices, limitations, and how to communicate methods to stakeholders.
Read how Effective Data Analysis explains hypothesis testing →
Causal AI
A hypothesis test is a statistical procedure for evaluating whether observed data provide sufficient evidence against a specified null hypothesis.
- How it is used
- Tests observed group differences, DAG-implied conditional independences, causal effect estimates, and assumptions behind those estimates.
- How deep it goes
- Works through null hypotheses, statistics, significance thresholds, and Python tests, while treating rigorous independence-as-alternative testing as mathematically nontrivial.
- Examples
- Uses transportation-survey chi-squared tests and an online-game comparison of purchases between high- and low-engagement players.
- What you can do after
- Apply it via pgmpy to challenge DAG independences and via DoWhy to test whether simulated assumption violations refute an estimate.
Statistics Every Programmer Needs
A hypothesis test is a statistical procedure for evaluating evidence against a specified null hypothesis.
- How it is used
- Uses statistical tests to assess evidence against a null hypothesis, including checks of variable normality before simple regression.
- How deep it goes
- Defines p-values and significance decisions, and discusses the conventional threshold and its false-positive and false-negative trade-off.
- Examples
- Runs Shapiro–Wilk tests on stage1 and stage2 variables and reports each test statistic and p-value.
- What you can do after
- Evaluate evidence by comparing a test's p-value with a predetermined threshold and deciding whether to reject or fail to reject the null.
Read how Statistics Every Programmer Needs explains hypothesis testing →
Grokking Bayes
A hypothesis test is a statistical procedure that evaluates whether observed data provide sufficient evidence against a specified null hypothesis.
- How it is used
- Frames observed-data questions as formal comparisons against a null model or between competing hypotheses.
- How deep it goes
- Explains null hypotheses, test statistics, tail probabilities, significance thresholds, binomial tests, and two-sample t-tests.
- Examples
- Uses a website redesign, 500 visitors, 63 conversions, a 10% baseline, and a Python binomial-test listing.
- What you can do after
- Apply a binomial test, compute a p-value, and make a threshold-based decision about evidence against a null hypothesis.
Machine Learning System Design
A statistical procedure for evaluating whether observed data provide evidence against a specified hypothesis or reference distribution.
- How it is used
- Uses statistical tests to decide whether A/B results indicate an effect and to detect drift during model monitoring.
- How deep it goes
- Defines hypotheses, p-values, and significance, then outlines simulation checks; defers detailed guidance on choosing a test.
- Examples
- Uses unequal-variance group comparisons, repeated simulated experiments, and drift detection as applications.
- What you can do after
- Evaluate A/B differences against a null hypothesis and validate test sensitivity through simulation.
Read how Machine Learning System Design explains hypothesis testing →
Designing AI Agents
An analytical technique that evaluates candidate explanations or predictions against evidence or observations.
- How it is used
- Turns a suspected Argus fault into a bounded test plan using supplied sandbox checks.
- How deep it goes
- Constrains investigation to five iterations and sandbox checks; confirming a fault does not authorize recovery.
- Examples
- Uses Argus to investigate whether a stated suspicion is a real bug.
- What you can do after
- Diagnose/debug with it by testing a suspicion through bounded sandbox checks.
Also covered in
- Statistics Slam Dunk
- The Well-Grounded Data Analyst
- Machine Learning Platform Engineering
- First-Party Data Activation
- Data-Driven Cybersecurity
- AI Agents in Action
What to know first
- categorical dataCategorical data consists of values representing membership in discrete categories rather than measured numerical magnitudes.
- null hypothesisA null hypothesis is a statistical claim about a population or data-generating process that serves as a reference assumption and is evaluated by a hypothesis test, often asserting no effect or difference.
- p-valueA p-value is the probability, assuming a specified null hypothesis, of obtaining a test statistic or result at least as extreme as the observed one.
- varianceVariance is the expected squared deviation of a random variable from its mean and measures the dispersion of its values.
Related concepts
- t-testincludes · A t-test is a statistical hypothesis test that uses a t-distributed statistic to assess a mean or difference in means under specified assumptions.
- chi-square testincludes · A chi-square test is a statistical hypothesis test that uses a chi-square-distributed statistic to evaluate hypotheses about categorical data, goodness of fit, or variance.
- posterior probabilitycontrasts with · A posterior probability or distribution represents the probability of an event, hypothesis, or unknown quantity conditional on observed evidence and prior information.
- p-valueuses · A p-value is the probability, assuming a specified null hypothesis, of obtaining a test statistic or result at least as extreme as the observed one.
- null hypothesisuses · A null hypothesis is a statistical claim about a population or data-generating process that serves as a reference assumption and is evaluated by a hypothesis test, often asserting no effect or difference.
- categorical dataused by · Categorical data consists of values representing membership in discrete categories rather than measured numerical magnitudes.
Questions about hypothesis testing
What is hypothesis testing?
Hypothesis testing is a formal statistical procedure for evaluating evidence from observed data about a specified hypothesis, often a null hypothesis.
Which Manning books explain hypothesis testing?
Timeless Algorithms, The Foundational Ideas, Grokking Statistics, Effective Data Analysis, Causal AI, Statistics Every Programmer Needs, Grokking Bayes, Machine Learning System Design and Designing AI Agents.
What should I know before learning hypothesis testing?
Manning books that teach hypothesis testing usually introduce categorical data, null hypothesis, p-value and variance first.
What is the difference between hypothesis testing and posterior probability?
Hypothesis testing is a formal statistical procedure for evaluating evidence from observed data about a specified hypothesis, often a null hypothesis. A posterior probability or distribution represents the probability of an event, hypothesis, or unknown quantity conditional on observed evidence and prior information.
What kinds of hypothesis testing are there?
Kinds of hypothesis testing include t-test and chi-square test.
What is hypothesis testing also called?
Hypothesis testing is also called statistical test, hypothesis test, significance testing or statistical significance test.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.







