A replicability crisis occurs when researchers cannot consistently reproduce published findings. P-hacking—selecting data, variables, subgroups, model parameters, or random seeds to obtain a desired result—can help create this problem. section 9.14
Common warning signs and safeguards include:
- Collecting only enough data to achieve a preferred result or removing inconvenient observations without justification. section 9.14
- Holding out a validation dataset to test a model on data untouched during iterative testing. section 9.14
- Recognizing confirmation bias, seeking contrary evidence, and gathering new data that could challenge the finding. section 9.14
When an outlier may be legitimate to remove
A sleep log includes zero hours after a 16-hour overnight drive. Excluding that unusual event may be reasonable when measuring normal sleep habits, but removing observations merely to improve a result is suspect. section 9.14
