Statistical significance
Statistical significance is a judgment, under a specified statistical test and criterion, that observed data are sufficiently inconsistent with a null hypothesis.
Statistics Slam Dunk
Statistical significance is an inferential assessment that an observed result is unlikely under a specified null hypothesis at a chosen significance threshold.
- How it is used
- Tests whether observed NBA group differences are plausibly due to chance before drawing analytical conclusions.
- How deep it goes
- Defines significance through p-values, thresholds, means, variances, and record counts, then applies Welch two-sample t-tests without deriving their mathematics.
- Examples
- Uses home-versus-road fouls and free throws, plus championship-team salary Gini coefficients and standardized scoring and defense boxplots.
- What you can do after
- Apply it via R’s t.test() to decide whether two-group mean differences meet a 5% significance threshold.
Read how Statistics Slam Dunk explains statistical significance →
Effective Data Analysis
A property of a statistical result indicating that it would be relatively unlikely under a specified null hypothesis according to a chosen significance criterion.
- How it is used
- Uses significance to interpret statistical-test results and communicate findings that support, contradict, or do not establish stakeholder hypotheses.
- How deep it goes
- Explains coefficients, degrees of freedom, thresholds, sample-size effects, and assumption checks, without treating a coefficient alone as decisive.
- Examples
- Illustrates a test-score comparison that becomes significant with a tenfold sample-size increase and stakeholder responses to non-significant findings.
- What you can do after
- Evaluate/understand test results with threshold and assumption checks, then communicate non-significant or conflicting findings without overstating them.
Read how Effective Data Analysis explains statistical significance →
Statistics Every Programmer Needs
Statistical significance is an assessment of whether observed evidence is sufficiently inconsistent with a specified null hypothesis under a statistical model.
- How it is used
- Uses hypothesis tests and regression analysis to decide whether observed associations are unlikely under a null hypothesis.
- How deep it goes
- Defines p-values, thresholds, and interpretation limits, then stops short of measuring effect size or proving a null hypothesis.
- Examples
- Uses Shapiro–Wilk tests on stage1 and stage2 data, reporting p-values of 0.2933 and 0.1479.
- What you can do after
- Evaluate test results against a predetermined threshold while separating statistical significance from effect size and hypothesis truth.
Read how Statistics Every Programmer Needs explains statistical significance →
The Well-Grounded Data Analyst
Statistical significance is the degree to which an observed result is unlikely under a specified null hypothesis according to a statistical test.
- How it is used
- Evaluates stakeholder hypotheses about associations in survey data and supplies cautious evidence for or against them.
- How deep it goes
- Explains null-hypothesis testing through p-values and rank-correlation code, but stops short of mathematical derivations.
- Examples
- Uses developer experience, AI-output trust, heatmaps, Spearman’s correlation, Kendall’s tau, and Python test output.
- What you can do after
- Evaluate/understand whether an observed survey association has evidence against its null hypothesis and communicate its strength cautiously.
Read how The Well-Grounded Data Analyst explains statistical significance →
Starting Data Analytics with Generative AI and Python
The degree to which observed data provide evidence against a specified null hypothesis under a statistical testing procedure.
- How it is used
- Supports cautious interpretation of statistical tests within business analytics and analytical design.
- How deep it goes
- Distinguishes statistical evidence from business meaning but does not develop testing procedures or mathematical details.
- Examples
- Uses correlation analysis and regression interpretation to contrast statistical significance with meaningful business relationships and predictive power.
- What you can do after
- Evaluate analytical findings by separating statistical evidence from practical importance and checking them against domain expectations.
Read how Starting Data Analytics with Generative AI and Python explains statistical significance →
Grokking Statistics
A result property indicating that an observed pattern would be relatively unlikely under a specified null model.
- How it is used
- Evaluates study and workplace claims while prompting scrutiny of bias, assumptions, alternative explanations, and incentives.
- How deep it goes
- Treats statistical significance as an interpretive claim to question, without supplying calculation or testing procedures.
- Examples
- Uses a pricing-system revenue claim and a cereal diabetes study to expose misleading interpretation and pressure.
- What you can do after
- Evaluate statistical claims by checking alternative explanations, ignored variables, outlier removal, and incentives.
Read how Grokking Statistics explains statistical significance →
Machine Learning System Design
A statistical procedure that evaluates whether observed data provide sufficient evidence against a null hypothesis according to a chosen significance criterion.
- How it is used
- Evaluates A/B-test differences against a null hypothesis before supporting rollout or further experimentation.
- How deep it goes
- Defines p-values, significance levels, hypotheses, and test choice, but defers detailed guidance for selecting the appropriate statistical test.
- Examples
- Uses Welch’s t-test and an A/B reporting table comparing conversion rate and average order value.
- What you can do after
- Evaluate A/B-test results and report significance alongside effect size, confidence intervals, practical metrics, and segment effects.
Read how Machine Learning System Design explains statistical significance →
Related concepts
- practical importancecontrasts with · Practical importance is the real-world significance of an effect, assessed by whether its magnitude meaningfully affects outcomes or decisions.
- p-hackingcontrasts with · P-hacking is manipulating analyses, data selection, testing procedures, stopping rules, or reporting choices to obtain apparently statistically significant results.
- p-valuemeasured by · A p-value is the probability, assuming a specified null hypothesis, of obtaining a test statistic or result at least as extreme as the observed one.
- t-testmeasured by · A t-test is a statistical hypothesis test that uses a t-distributed statistic to assess a mean or difference in means under specified assumptions.
- correlationcontrasts with · Correlation is a statistical relationship describing how two or more variables or quantities vary together, including the strength and direction of their association, without by itself implying causation.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.