The multiple-comparisons problem arises when several groups, hypotheses, or pairwise contrasts are tested. Each additional comparison increases the chance of a false-positive Type 1 result, even when corrected p-values are used. section 5.3.1 section 5.4
Reduce the risk by:
- Choosing statistical tests that fit the data and checking their assumptions before testing. section 5.3.1
- Keeping the number of groups and interactions under control; a suggested two-factor ANOVA design uses two groups for one variable and three for another. section 5.3.1
- Using post hoc tests after an omnibus result involving three or more groups, because these tests correct comparisons while identifying which groups differ. section 5.4
- Report non-significant results and avoid treating a low p-value as sufficient evidence when many comparisons were made. section 5.3.1 section 5.4
Figure 5.16 I implore you not to run excessive comparisons, even with a sufficiently large sample size.
Use the figure to visualize why excessive comparisons can produce apparently significant results, even with a large sample. section 5.3.1
