A p-value estimates how likely a test statistic at least as extreme as the observed one would be if the null hypothesis were true. In a comparison of groups, it helps assess whether an observed difference is statistically significant. It does not, by itself, establish that the difference matters in practice. section 4.4
Interpret a p-value alongside the test design:
- Choose an alpha level—the threshold for statistical significance—before comparing it with the p-value. A test coefficient alone is not enough to interpret a result. section 4.2.1
- Check the test’s assumptions and sample size. Parametric tests make assumptions about the population distribution, and an inappropriate sample size can make a result misleading even when other assumptions hold. section 4.2.1 section 4.4
- Report the finding and its limitations precisely, especially to stakeholders who may otherwise treat statistical significance as the whole conclusion. section 4.4
Table 4.1 Abbreviated t-distribution table showing that increasing the degrees of freedom has diminishing returns on the t-critical values at each alpha level (p-value threshold).
|
degrees of freedom
|
alpha level
|
||||
|---|---|---|---|---|---|
|
0.1
|
0.05
|
0.01
|
0.005
|
0.001
|
|
|
10
|
1.81
|
2.23
|
3.17
|
3.58
|
4.59
|
|
20
|
1.72
|
2.09
|
2.85
|
3.15
|
3.85
|
|
30
|
1.70
|
2.04
|
2.75
|
3.03
|
3.65
|
|
40
|
1.68
|
2.02
|
2.70
|
2.97
|
3.55
|
|
50
|
1.68
|
2.01
|
2.68
|
2.94
|
3.50
|
|
60
|
1.67
|
2.00
|
2.66
|
2.91
|
3.46
|
|
70
|
1.67
|
1.99
|
2.65
|
2.90
|
3.44
|
|
80
|
1.66
|
1.99
|
2.64
|
2.89
|
3.42
|
|
90
|
1.66
|
1.99
|
2.63
|
2.88
|
3.40
|
|
100
|
1.66
|
1.98
|
2.63
|
2.87
|
3.39
|
|
150
|
1.66
|
1.98
|
2.61
|
2.85
|
3.36
|
|
200
|
1.65
|
1.97
|
2.60
|
2.84
|
3.34
|
The t-distribution table pairs degrees of freedom with alpha levels to identify critical t-values. It also shows that increasing degrees of freedom has diminishing effects on those critical values. section 4.2.1
Changing the threshold and sample size
Plan the sample size for an independent-samples t-test with statsmodels, then run the test on generated normal samples with SciPy. Repeat at alpha levels of 0.05 and 0.01 using their recommended sample sizes. Compare the resulting t-values and p-values rather than assuming that either stays fixed when the design changes. section 4.2.2
P-values also appear in non-parametric comparisons. A Kruskal–Wallis exercise varies sample size and compares p-values from different post hoc tests, reinforcing the need to identify which test produced each result. section 5.2.2
