P-hacking
P-hacking is manipulating analyses, data selection, testing procedures, stopping rules, or reporting choices to obtain apparently statistically significant results.
Effective Data Analysis
P-hacking is the manipulation of analyses, data collection, or reporting choices to obtain a statistically significant p-value.
- How it is used
- Frames p-hacking as an analyst responsibility when making population-level inferences from statistical tests.
- How deep it goes
- Explains repeated testing and analytical flexibility, but does not detail a complete prevention workflow.
- Examples
- Uses an app-layout test, repeated statistical tests, outlier removal, transformations, and an indefinite-testing figure.
- What you can do after
- Diagnose/debug with it by recognizing analytical choices that can manufacture statistical significance and misrepresent population findings.
Grokking Statistics
P-hacking is the selective analysis or threshold manipulation of data to obtain or present a statistically significant result.
- How it is used
- Identifies how flexible analysis can be bent toward statistically significant or otherwise favorable findings.
- How deep it goes
- Explains the definition, common modeling behaviors, test-set leakage, incentives, and basic mitigation without formal statistical derivations.
- Examples
- Uses selective data collection, unjustified outlier removal, subgroup shopping, and repeated model tuning against a test set.
- What you can do after
- Diagnose p-hacking by recognizing cherry-picked analyses, test-set leakage, and pressures favoring results, then re-evaluate for contrary evidence.
Experimentation for Engineers
The misuse of statistical analysis or stopping rules to make results appear statistically significant when the apparent effect is not real.
- How it is used
- Guards A/B-test acceptance decisions against apparently significant results produced by unsuitable stopping or metric-selection practices.
- How deep it goes
- Explains early stopping, z-score thresholds, operational failure, and family-wise error, then stops short of correction methods.
- Examples
- Uses simulated uniform profits, monitored z scores, an apparent win at n=30, and false-positive-rate comparisons.
- What you can do after
- Diagnose how repeated monitoring or metric searching can turn random variation into an accepted A/B-test win.
Related concepts
- early stoppingincludes · Early stopping is a technique that terminates an iterative process before its planned completion when a specified condition indicates that further work is unnecessary or unlikely to improve the objective.
- hypothesis testinguses · Hypothesis testing is a formal statistical procedure that uses observed data to evaluate evidence about a specified hypothesis, often a null hypothesis.
- statistical significancecontrasts with · Statistical significance is a judgment, under a specified statistical test and criterion, that observed data are sufficiently inconsistent with a null hypothesis.
- evaluation datasetuses · An evaluation dataset is a collection of examples or test cases, often with expected or judged outcomes, designated to assess a model or system against defined criteria.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.