liveBook · Manning

P-hacking

As explained in Timeless Algorithms, The Foundational Ideas

P-hacking is the practice of selectively changing analyses or reporting choices to obtain apparently statistically significant results.

P-hacking changes significance thresholds, tests multiple hypotheses until one appears significant, or selectively reports favorable results after examining the data. These choices increase the likelihood of false discoveries and weaken statistical inference. Prespecifying the decision rule helps keep conclusions from being shaped by the observed results. section 4.2.3

A borderline clinical-trial result

A p-value of 0.057 does not establish that a drug is ineffective or that the null hypothesis is true; it falls short of the predetermined standard for rejecting the null. Relaxing the significance threshold after seeing this result would be p-hacking. Report the exact p-value and treat the finding as suggestive, warranting a larger, adequately powered follow-up study. section 4.2.3

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Names three p-hacking practices and their false-discovery risk without quantifying that risk.
Examples
Uses a clinical-trial p-value of 0.057 to illustrate a borderline result and the temptation to relax a threshold.
What it assumes
Assumes familiarity with p-values, significance levels, and reject-or-fail-to-reject decisions rather than defining them from zero.
What you can do after
Evaluate borderline results by reporting exact p-values and treating suggestive findings as grounds for adequately powered follow-up.
How it connects
Connects p-hacking to hypothesis testing, significance levels, p-values, statistical power, and false discoveries.

Also explained in

Compare how 4 books explain p-hacking →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.