P-value
A p-value is the probability, assuming a specified null hypothesis, of obtaining a test statistic or result at least as extreme as the observed one.
Statistics Slam Dunk
A p-value is the probability, under a specified null hypothesis, of obtaining a result at least as extreme as the observed result according to a test statistic.
- How it is used
- Supports null-hypothesis decisions in normality tests, t-tests, correlation tests, ANOVAs, and regression modeling.
- How deep it goes
- Defines threshold-based interpretation and demonstrates it across R test outputs, without deriving p-value calculations.
- Examples
- Uses NBA foul calls, free-throw attempts, salary and win-share Gini coefficients, normality checks, and wins models.
- What you can do after
- Apply it via base R tests and model summaries to reject or fail to reject a stated null hypothesis.
Statistics Every Programmer Needs
A p-value is the probability, under a specified null hypothesis, of obtaining a result at least as extreme as the observed result.
- How it is used
- Guides null-hypothesis decisions about regression models, individual predictors, and observed-versus-expected counts.
- How deep it goes
- Defines the conditional probability, works through threshold decisions, and demonstrates a chi-square computation without deriving a general p-value formula.
- Examples
- Uses runner-time regression, raisin-variety classification, and a SciPy test of world-population leading digits against Benford’s law.
- What you can do after
- Apply it via SciPy and statsmodels to interpret significance while checking fit, assumptions, and predictive performance separately.
Read how Statistics Every Programmer Needs explains p-value →
Grokking Statistics
A p-value is a probability, calculated under the null hypothesis, of obtaining a result at least as extreme as the observed result.
- How it is used
- Uses p-values to judge whether sample evidence challenges a null hypothesis in mean and correlation tests.
- How deep it goes
- Defines tail-area meaning, contrasts one- and two-tailed calculations, and applies decision rules without deriving proofs.
- Examples
- Uses tea identification, potato chip weights, and sports drink sales versus temperature datasets.
- What you can do after
- Apply it via SciPy to calculate p-values and decide whether to reject a specified null hypothesis.
Effective Data Analysis
A p-value is the probability of obtaining a test statistic at least as extreme as the observed value assuming the null hypothesis is true.
- How it is used
- Uses p-values to interpret test statistics and assess statistical significance in analytics.
- How deep it goes
- Explains significance thresholds and sample-size limitations, then provides exercises varying alpha and sample size.
- Examples
- Uses student test scores and simulated independent samples to explore significance and sample size.
- What you can do after
- Practice interpreting p-values by running independent-samples t-tests and comparing results across thresholds and sample sizes.
Timeless Algorithms, The Foundational Ideas
A p-value is the probability, under a null hypothesis, of observing a result at least as extreme as the one obtained.
- How it is used
- Turns sample evidence into a hypothesis-testing decision against a predetermined standard.
- How deep it goes
- Defines the null-based probability and works through decision thresholds and interpretation, including a near-cutoff result.
- Examples
- Uses a small clinical trial and beverage fill measurements to illustrate one- and two-tailed decisions.
- What you can do after
- Evaluate evidence against a predetermined significance level without treating the p-value as the probability that the null hypothesis is true.
Read how Timeless Algorithms, The Foundational Ideas explains p-value →
Grokking Bayes
A p-value is the probability, under a null hypothesis, of obtaining data at least as extreme as the observed data according to a specified test statistic.
- How it is used
- Evaluates how surprising observed data are under a null hypothesis in classical hypothesis testing.
- How deep it goes
- Defines null-based extremeness, test statistics, tail probabilities, and threshold decisions, but does not estimate hypothesis probabilities.
- Examples
- Uses a website conversion test with 500 visitors, 63 conversions, a 0.1 baseline, a right tail, and scipy.stats.binomtest.
- What you can do after
- Apply it via a classical hypothesis test to quantify null-based extremeness and make a threshold-based rejection decision.
Fight Fraud with Machine Learning
A p-value is the probability, under a statistical test's null hypothesis, of obtaining a result at least as extreme as the observed result.
- How it is used
- Uses statistical significance to qualify ANOVA-based feature rankings for phishing-fraud detection.
- How deep it goes
- Defines a practical significance threshold and inspects returned values, but does not derive p-values or test assumptions.
- Examples
- Uses phishing-page features, including NumNumericChars and PctExtResourceUrls, with scikit-learn p-value output.
- What you can do after
- Evaluate whether ANOVA F-scores are statistically significant enough to support numerical-feature analysis.
Read how Fight Fraud with Machine Learning explains p-value →
Causal AI
A p-value is the probability, under a null hypothesis, of obtaining a test statistic at least as extreme as the observed one.
- How it is used
- Uses p-values to assess statistical evidence for dependence or conditional independence while building and checking causal models.
- How deep it goes
- Explains the definition, threshold rule, bootstrap workflow, and sample-size limitation, but stops short of deriving test statistics.
- Examples
- Uses E ⊥ T | O, R, a 30-point dataset, bootstrap resampling, confidence intervals, and plots of p-values against data size.
- What you can do after
- Diagnose/debug conditional-independence tests by comparing thresholds, bootstrapping across sample sizes, and interpreting results cautiously.
LLM Evaluation and Alignment, The Foundational Ideas
A p-value is the probability, under a specified null hypothesis, of obtaining a result at least as extreme as the observed result.
- How it is used
- Evaluates whether an observed correlation is unlikely under a no-relationship null hypothesis.
- How deep it goes
- Explains significance, computation, and interpretation, but stops short of treating significance as practical correlation strength.
- Examples
- Uses human and metric scores, bootstrap resampling, and contrasting p-values to illustrate evaluation decisions.
- What you can do after
- Evaluate metric evidence by interpreting p-values alongside correlation magnitudes and confidence intervals.
Read how LLM Evaluation and Alignment, The Foundational Ideas explains p-value →
Also covered in
- Machine Learning System Design
- Starting Data Analytics with Generative AI and Python
- The Well-Grounded Data Analyst
- Designing AI SystemsIn early access (MEAP)
- AI Model EvaluationIn early access (MEAP)
What to know first
- null hypothesisA null hypothesis is a statistical claim about a population or data-generating process that serves as a reference assumption and is evaluated by a hypothesis test, often asserting no effect or difference.
- varianceVariance is the expected squared deviation of a random variable from its mean and measures the dispersion of its values.
- meanThe mean is a measure of central location equal to the arithmetic average of numerical observations, or the probability-weighted average of a random variable’s possible values.
- anomalyAn anomaly is an observation, data state, or behavior that substantially deviates from an expected or prevailing pattern or condition.
Related concepts
- bayes factoralternative to · A Bayes factor is the ratio of the marginal likelihoods of two competing models or hypotheses, quantifying their relative support from observed data.
- correlation testingproduced by · Correlation testing is a statistical procedure for evaluating the association between variables, often against a specified null value.
- Shapiro–Wilk testproduced by · The Shapiro–Wilk test is a statistical hypothesis test assessing whether a sample is consistent with a normal distribution.
- hypothesis testingused by · Hypothesis testing is a formal statistical procedure for evaluating evidence from observed data about a specified hypothesis, often a null hypothesis.
Questions about p-value
What is p-value?
A p-value is the probability, assuming a specified null hypothesis, of obtaining a test statistic or result at least as extreme as the observed one.
Which Manning books explain p-value?
Statistics Slam Dunk, Statistics Every Programmer Needs, Grokking Statistics, Effective Data Analysis, Timeless Algorithms, The Foundational Ideas, Grokking Bayes, Fight Fraud with Machine Learning, Causal AI and 1 more.
What should I know before learning p-value?
Manning books that teach p-value usually introduce null hypothesis, variance, mean and anomaly first.
What is the difference between p-value and bayes factor?
A p-value is the probability, assuming a specified null hypothesis, of obtaining a test statistic or result at least as extreme as the observed one. A Bayes factor is the ratio of the marginal likelihoods of two competing models or hypotheses, quantifying their relative support from observed data.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.








