liveBook · Manning

Statistical significance

As explained in Statistics Every Programmer Needs

Statistical significance is an assessment of whether observed evidence is sufficiently inconsistent with a specified null hypothesis under a statistical model.

Statistical significance assesses whether observed results are sufficiently unlikely under a null hypothesis to support rejecting it. A p-value measures how likely the observed results, or more extreme results, would be if the null hypothesis were true. section 4.2.1

Interpret statistical significance with a predetermined threshold and alongside model assumptions:

  • Compare the p-value with a chosen significance level; p < 0.05 conventionally counts as statistically significant and leads to rejecting the null hypothesis. section 4.2.1
  • Treat 5% as a convention rather than a universal principle; another predetermined level can provide the decision rule. section 4.2.1
  • Do not interpret significance as the probability that the null hypothesis is true or false, or as a measure of effect size. section 4.2.1
  • Use regression significance to identify influential predictors, while checking collinearity and other model assumptions. section 4.0

Reading a Shapiro–Wilk test

A Python analysis reports p-values of 0.2933 for stage1 and 0.1479 for stage2. These outputs illustrate how a statistical test supplies p-values for comparison with a chosen significance threshold. section 4.2.1

>>> from scipy import stats

>>> print(stats.shapiro(mds.stage1))
ShapiroResult(statistic=0.9446673217389622, pvalue=0.29328758849732417)

>>> print(stats.shapiro(mds.stage2))
ShapiroResult(statistic=0.9290259439587454, pvalue=0.1478907551379664)

Inspect the paired Shapiro–Wilk test outputs and their p-values. section 4.2.1

How Statistics Every Programmer Needs covers it

How deep it goes
Defines p-values, thresholds, and interpretation limits, then stops short of measuring effect size or proving a null hypothesis.
Examples
Uses Shapiro–Wilk tests on stage1 and stage2 data, reporting p-values of 0.2933 and 0.1479.
What it assumes
Starts with null hypotheses, p-values, significance thresholds, and regression predictors rather than assuming them.
What you can do after
Evaluate test results against a predetermined threshold while separating statistical significance from effect size and hypothesis truth.
How it connects
Connects statistical tests, null hypotheses, p-values, linear regression, collinearity, Type I errors, and Type II errors.

Also explained in

Compare how 7 books explain statistical significance →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Statistics Every Programmer Needs. The section links point to the parts of the book it draws on.