liveBook · Manning

Model torturing

As explained in Grokking Statistics

Repeated or selective modification of a statistical model until it produces a preferred result.

Model torturing means repeatedly or selectively modifying a statistical model until it produces a preferred result. It is a form of statistical abuse: the final analysis may look statistically validated while hiding biased choices. section 1.4

When the answer is chosen first

A researcher may be pressured to use a favorable dataset to impress investors. Similar pressures appear when leaders want research supporting a cause or campaign, or when financial ties encourage reassuring claims about sugary drinks. These incentives can steer model choices toward a desired conclusion rather than a trustworthy one. section 1.4

The practical defense is critical thinking: look beyond statistical validation and ask whether confirmation bias, data choices, or model changes may have shaped the outcome. section 1.4

How Grokking Statistics covers it

How deep it goes
Defines the abuse and names its biases, but defers detailed treatment of how model torturing works.
Examples
Uses favorable datasets, cause-driven research, campaign-supporting research, and industry-influenced health claims as warning examples.
What it assumes
Assumes some statistical proficiency while emphasizing critical thinking about how results are produced.
What you can do after
Evaluate statistical claims by looking for biased incentives and selective choices behind apparently validated results.
How it connects
Connects model torturing with confirmation bias, data torturing, statistical validation, and critical thinking.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Statistics. The section links point to the parts of the book it draws on.