P-hacking means shopping among models, variables, data choices, or analysis decisions until a desired result appears—often a low p-value or statistically significant finding. It can be deliberate, but it often begins as an innocent attempt to make a model work and gradually turns into confirmation bias. section 9.12
Common forms of p-hacking include:
- Collecting only enough data to obtain the desired result. section 9.14
- Removing inconvenient observations as outliers or noise without justification. section 9.14
- Shopping for variables, model parameters, or subgroups that produce favorable findings. section 9.14
- Repeating train/test splits and model changes until one performs well on the test set. section 9.14
Torturing a test set
A practitioner repeatedly changes the model and data after inspecting test performance. A strong final score may look convincing, but the test set has leaked clues into the training process. Holding out a separate validation dataset provides an additional check on data that was never part of that iterative testing. section 9.14
The practical defense is self-awareness: notice when the analysis is confirming a preferred answer, step back, reconsider the approach, and remain willing to find contrary data. Pressure from research funding, jobs, clients, or startup demonstrations can make cherry-picking especially tempting. section 9.12 section 9.15
