P-hacking
P-hacking is manipulating data collection, analysis, or reporting choices to obtain or present an apparently statistically significant result.
Effective Data Analysis
P-hacking is the manipulation of analyses, data collection, or reporting choices to obtain a statistically significant p-value.
- How it is used
- Frames p-hacking as an analyst responsibility when making population-level inferences from statistical tests.
- How deep it goes
- Explains repeated testing and analytical flexibility, but does not detail a complete prevention workflow.
- Examples
- Uses an app-layout test, repeated statistical tests, outlier removal, transformations, and an indefinite-testing figure.
- What you can do after
- Diagnose/debug with it by recognizing analytical choices that can manufacture statistical significance and misrepresent population findings.
Grokking Statistics
P-hacking is the selective analysis or threshold manipulation of data to obtain or present a statistically significant result.
- How it is used
- Identifies how flexible analysis can be bent toward statistically significant or otherwise favorable findings.
- How deep it goes
- Explains the definition, common modeling behaviors, test-set leakage, incentives, and basic mitigation without formal statistical derivations.
- Examples
- Uses selective data collection, unjustified outlier removal, subgroup shopping, and repeated model tuning against a test set.
- What you can do after
- Diagnose p-hacking by recognizing cherry-picked analyses, test-set leakage, and pressures favoring results, then re-evaluate for contrary evidence.
Experimentation for Engineers
The misuse of statistical analysis or stopping rules to make results appear statistically significant when the apparent effect is not real.
- How it is used
- Guards A/B-test acceptance decisions against apparently significant results produced by unsuitable stopping or metric-selection practices.
- How deep it goes
- Explains early stopping, z-score thresholds, operational failure, and family-wise error, then stops short of correction methods.
- Examples
- Uses simulated uniform profits, monitored z scores, an apparent win at n=30, and false-positive-rate comparisons.
- What you can do after
- Diagnose how repeated monitoring or metric searching can turn random variation into an accepted A/B-test win.
Timeless Algorithms, The Foundational Ideas
P-hacking is the practice of selectively changing analyses or reporting choices to obtain apparently statistically significant results.
- How it is used
- Protects hypothesis-test decisions from post hoc changes to thresholds, hypotheses, and reporting.
- How deep it goes
- Names three p-hacking practices and their false-discovery risk without quantifying that risk.
- Examples
- Uses a clinical-trial p-value of 0.057 to illustrate a borderline result and the temptation to relax a threshold.
- What you can do after
- Evaluate borderline results by reporting exact p-values and treating suggestive findings as grounds for adequately powered follow-up.
Read how Timeless Algorithms, The Foundational Ideas explains p-hacking →
Also covered in
- Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI
- Build AI Drug Discovery PipelinesIn early access (MEAP)
- Grokking BayesIn early access (MEAP)
What to know first
- p-valueA p-value is the probability, assuming a specified null hypothesis, of obtaining a test statistic or result at least as extreme as the observed one.
- false positiveA false positive is a positive result that incorrectly indicates a target condition is present when it is absent.
- data transformationA data transformation is an operation that maps input data or representations to output by changing their values, structure, or form.
- normal distributionA normal distribution is a continuous probability distribution with a symmetric, bell-shaped density determined by its mean and variance.
Related concepts
- early stoppingincludes · Early stopping is a technique that terminates an iterative process before its planned completion when a specified condition indicates that further work is unnecessary or unlikely to improve the objective.
- hypothesis testinguses · Hypothesis testing is a formal statistical procedure for evaluating evidence from observed data about a specified hypothesis, often a null hypothesis.
- evaluation datasetuses · An evaluation dataset is a collection of data or test cases used to assess a model or system against specified outcomes or criteria.
Questions about p-hacking
What is p-hacking?
P-hacking is manipulating data collection, analysis, or reporting choices to obtain or present an apparently statistically significant result.
Which Manning books explain p-hacking?
Effective Data Analysis, Grokking Statistics, Experimentation for Engineers and Timeless Algorithms, The Foundational Ideas.
What should I know before learning p-hacking?
Manning books that teach p-hacking usually introduce p-value, false positive, data transformation and normal distribution first.
What is the difference between p-hacking and classical hypothesis testing?
P-hacking is manipulating data collection, analysis, or reporting choices to obtain or present an apparently statistically significant result. Classical hypothesis testing is a frequentist procedure that evaluates evidence against a null hypothesis using the sampling distribution of a test statistic.
What kinds of p-hacking are there?
Kinds of p-hacking include early stopping and family-wise error.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.



