P-hacking, or data fishing, manipulates data, hypotheses, designs, tests, outlier decisions, or transformations until a statistically significant p-value appears. The resulting finding may not reflect the population. section 5.3.2
The central risk is inflated false-positive evidence: at an alpha level of 0.05, repeatedly testing increases the chance that at least one test rejects the null hypothesis and commits a Type 1 error. section 5.3.2
Common forms of p-hacking include:
- Running large or indefinite permutations of statistical tests until one produces significance. section 5.3.2
- Continually changing hypotheses, analytical designs, outlier-removal methods, or data transformations until the result crosses the significance threshold. section 5.3.2
- Dropping individual observations to improve a result rather than applying a systematic rule for handling outliers. section 4.2.1
App layout and weekly visits
An analyst tests whether changing an app layout causes 10% more users to visit weekly. Trying many tests, designs, or data treatments until one supports that claim turns exploratory analysis into p-hacking. section 5.3.2
Figure 5.18 P-hacking involves testing a dataset indefinitely until something returns a significant value.
The figure depicts p-hacking as testing a dataset indefinitely until a significant value appears. section 5.3.2
