Statistical significance
Statistical significance is a judgment that observed data are sufficiently inconsistent with a specified null hypothesis under a statistical test and chosen criterion.
Statistics Slam Dunk
Statistical significance is an inferential assessment that an observed result is unlikely under a specified null hypothesis at a chosen significance threshold.
- How it is used
- Tests whether observed NBA group differences are plausibly due to chance before drawing analytical conclusions.
- How deep it goes
- Defines significance through p-values, thresholds, means, variances, and record counts, then applies Welch two-sample t-tests without deriving their mathematics.
- Examples
- Uses home-versus-road fouls and free throws, plus championship-team salary Gini coefficients and standardized scoring and defense boxplots.
- What you can do after
- Apply it via R’s t.test() to decide whether two-group mean differences meet a 5% significance threshold.
Read how Statistics Slam Dunk explains statistical significance →
Effective Data Analysis
A property of a statistical result indicating that it would be relatively unlikely under a specified null hypothesis according to a chosen significance criterion.
- How it is used
- Guides interpretation of test results and communication when findings conflict with stakeholder expectations.
- How deep it goes
- Explains test statistics, thresholds, degrees of freedom, assumption checks, and interpretation of contrasting outcomes.
- Examples
- Uses changing t-values and sample sizes to illustrate significance, alongside stakeholder hypotheses that results contradict or do not significantly support.
- What you can do after
- Evaluate and understand significance results while discussing uncertainty and measurement choices with stakeholders.
Read how Effective Data Analysis explains statistical significance →
Statistics Every Programmer Needs
Statistical significance is an assessment of whether observed evidence is sufficiently inconsistent with a specified null hypothesis under a statistical model.
- How it is used
- Uses hypothesis tests and regression analysis to decide whether observed associations are unlikely under a null hypothesis.
- How deep it goes
- Defines p-values, thresholds, and interpretation limits, then stops short of measuring effect size or proving a null hypothesis.
- Examples
- Uses Shapiro–Wilk tests on stage1 and stage2 data, reporting p-values of 0.2933 and 0.1479.
- What you can do after
- Evaluate test results against a predetermined threshold while separating statistical significance from effect size and hypothesis truth.
Read how Statistics Every Programmer Needs explains statistical significance →
The Well-Grounded Data Analyst
Statistical significance is the degree to which an observed result is unlikely under a specified null hypothesis according to a statistical test.
- How it is used
- Evaluates stakeholder hypotheses about associations in survey data and supplies cautious evidence for or against them.
- How deep it goes
- Explains null-hypothesis testing through p-values and rank-correlation code, but stops short of mathematical derivations.
- Examples
- Uses developer experience, AI-output trust, heatmaps, Spearman’s correlation, Kendall’s tau, and Python test output.
- What you can do after
- Evaluate/understand whether an observed survey association has evidence against its null hypothesis and communicate its strength cautiously.
Read how The Well-Grounded Data Analyst explains statistical significance →
Timeless Algorithms, The Foundational Ideas
Statistical significance is an assessment of whether an observed result is sufficiently unlikely under a specified null hypothesis.
- How it is used
- Connects hypothesis-test decisions to substantive conclusions without treating statistical significance as the sole measure of importance.
- How deep it goes
- Applies a predetermined threshold to an ambiguous trial result and distinguishes non-rejection from proof of the null.
- Examples
- Uses a small trial testing whether a drug raises recovery above a historical 50% baseline.
- What you can do after
- Evaluate conclusions by relating test decisions to assumptions, data quality, study scope, and practical stakes.
Read how Timeless Algorithms, The Foundational Ideas explains statistical significance →
Starting Data Analytics with Generative AI and Python
The degree to which observed data provide evidence against a specified null hypothesis under a statistical testing procedure.
- How it is used
- Supports cautious interpretation of statistical tests within business analytics and analytical design.
- How deep it goes
- Distinguishes statistical evidence from business meaning but does not develop testing procedures or mathematical details.
- Examples
- Uses correlation analysis and regression interpretation to contrast statistical significance with meaningful business relationships and predictive power.
- What you can do after
- Evaluate analytical findings by separating statistical evidence from practical importance and checking them against domain expectations.
Read how Starting Data Analytics with Generative AI and Python explains statistical significance →
Grokking Statistics
A result property indicating that an observed pattern would be relatively unlikely under a specified null model.
- How it is used
- Evaluates study and workplace claims while prompting scrutiny of bias, assumptions, alternative explanations, and incentives.
- How deep it goes
- Treats statistical significance as an interpretive claim to question, without supplying calculation or testing procedures.
- Examples
- Uses a pricing-system revenue claim and a cereal diabetes study to expose misleading interpretation and pressure.
- What you can do after
- Evaluate statistical claims by checking alternative explanations, ignored variables, outlier removal, and incentives.
Read how Grokking Statistics explains statistical significance →
LLM Evaluation and Alignment, The Foundational Ideas
Statistical significance is the condition in which an observed result is sufficiently unlikely under a null hypothesis according to a chosen threshold.
- How it is used
- Uses hypothesis testing to qualify evaluator–metric relationships during evaluation analysis.
- How deep it goes
- Defines p-values, thresholds, confidence intervals, and effect-size caveats, then stops short of broader hypothesis-testing procedures.
- Examples
- Contrasts small significant correlations with larger nonsignificant correlations and demonstrates bootstrap intervals for Spearman’s ρ.
- What you can do after
- Evaluate/understand whether an observed evaluation correlation supports a reliable claim, using p-values, effect sizes, and confidence intervals.
Read how LLM Evaluation and Alignment, The Foundational Ideas explains statistical significance →
Machine Learning System Design
A statistical procedure that evaluates whether observed data provide sufficient evidence against a null hypothesis according to a chosen significance criterion.
- How it is used
- Evaluates A/B-test differences against a null hypothesis before supporting rollout or further experimentation.
- How deep it goes
- Defines p-values, significance levels, hypotheses, and test choice, but defers detailed guidance for selecting the appropriate statistical test.
- Examples
- Uses Welch’s t-test and an A/B reporting table comparing conversion rate and average order value.
- What you can do after
- Evaluate A/B-test results and report significance alongside effect size, confidence intervals, practical metrics, and segment effects.
Read how Machine Learning System Design explains statistical significance →
Also covered in
- Designing AI SystemsIn early access (MEAP)
- Experimentation for Engineers
- First-Party Data Activation
- Data-Driven Cybersecurity
What to know first
- categorical dataCategorical data consists of values representing membership in discrete categories rather than measured numerical magnitudes.
- false positiveA false positive is a positive result that incorrectly indicates a target condition is present when it is absent.
- missing valueA missing value is an expected data value or observation that is absent, unavailable, unknown, or inapplicable.
- varianceVariance is the expected squared deviation of a random variable from its mean and measures the dispersion of its values.
Related concepts
- practical importancecontrasts with · Practical importance is the real-world significance of an effect, assessed by whether its magnitude meaningfully affects outcomes or decisions.
- p-valuemeasured by · A p-value is the probability, assuming a specified null hypothesis, of obtaining a test statistic or result at least as extreme as the observed one.
- correlationcontrasts with · Correlation is a statistical association describing how variables vary together in direction or degree, without by itself establishing causation.
- t-testmeasured by · A t-test is a statistical hypothesis test that uses a t-distributed statistic to assess a mean or difference in means under specified assumptions.
- effect sizecontrasts with · An effect size is a quantitative measure of the magnitude of a difference, association, relationship, or other observed phenomenon, independent of statistical significance.
Questions about statistical significance
What is statistical significance?
Statistical significance is a judgment that observed data are sufficiently inconsistent with a specified null hypothesis under a statistical test and chosen criterion.
Which Manning books explain statistical significance?
Statistics Slam Dunk, Effective Data Analysis, Statistics Every Programmer Needs, The Well-Grounded Data Analyst, Timeless Algorithms, The Foundational Ideas, Starting Data Analytics with Generative AI and Python, Grokking Statistics, LLM Evaluation and Alignment, The Foundational Ideas and 1 more.
What should I know before learning statistical significance?
Manning books that teach statistical significance usually introduce categorical data, false positive, missing value and variance first.
What is the difference between statistical significance and practical importance?
Statistical significance is a judgment that observed data are sufficiently inconsistent with a specified null hypothesis under a statistical test and chosen criterion. Practical importance is the real-world significance of an effect, assessed by whether its magnitude meaningfully affects outcomes or decisions.
What is statistical significance also called?
Statistical significance is also called statistically significant.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.








