Data torturing means repeatedly or selectively manipulating analyses until the data produce a desired finding—like rolling the dice until you get a Yahtzee. It can hide behind statistical validation while confirmation bias steers the outcome. section 1.4
The cereal-and-diabetes study
A cereal company receives evidence that its product may raise diabetes risk. Instead of changing the cereal, it pressures the researcher to re-evaluate the findings, search for alternative variables such as lifestyle, or bury the study. Selecting supportive studies while hiding unfavorable ones is another form of the same problem. section 1.4
Watch for data torturing when incentives encourage people to:
- Cherry-pick findings that support a preferred narrative. section 1.4
- Choose a favorable dataset or search among variables until a result looks compelling. section 1.4
- Treat statistical validation as proof while ignoring confirmation bias and competing evidence. section 1.4
