liveBook · Manning

Confirmation bias

As explained in The Coder Cafe

Confirmation bias is the tendency to favor information that supports an existing belief or hypothesis while discounting contradictory evidence.

Confirmation bias is the tendency to search for, interpret, favor, or recall information that supports an existing belief. In software work, it can turn a reasonable starting hypothesis into a wrong diagnosis. Starting with a suspected cause is not the problem; looking only for evidence that supports it is. section 9.2.2

A rollout that looked guilty

Suppose machine panics spike around a kernel rollout. The rollout is worth investigating, especially if similar changes have caused trouble before. But rolling it back without establishing a causal link lets the initial suspicion drive the response. Check whether panics rose precisely when the rollout began and whether the increase tracks the proportion of machines running the new kernel. If those signals do not support the hypothesis, look for another cause, including a lurking variable. section 7.7.4

Keep a hypothesis testable rather than letting it become the answer:

  • Write down the assumption so it stays visible. section 9.10
  • Seek evidence that could disprove it, not just dashboards that could confirm it. For a suspected request spike, also consider a recent code change or network issue. section 9.2.2
  • Be willing to abandon a hypothesis that does not explain the observations. section 7.8

The same failure can affect decisions outside incident response. An A/B test designed mainly around metrics favorable to a proposed feature change can miss evidence against it. The practical check is whether the evidence gathered could genuinely change the decision. section 9.2.2

How The Coder Cafe covers it

How deep it goes
Defines the bias and works through selective evidence gathering, ambiguous interpretation, and a kernel-rollout hypothesis.
Examples
Uses a suspected request spike, a kernel rollout, and an A/B test focused on favorable metrics.
What it assumes
Defines confirmation bias from scratch while using dashboards, incident response, and A/B tests as engineering contexts.
What you can do after
Diagnose/debug with it by recording assumptions, seeking disconfirming evidence, and dropping hypotheses that do not fit the observations.
How it connects
Connects confirmation bias to cognitive biases, incident response, A/B testing, and lurking variables.

Also explained in

Compare how 5 books explain confirmation bias →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of The Coder Cafe. The section links point to the parts of the book it draws on.