liveBook · Manning

Bayes' theorem

As explained in Timeless Algorithms, The Foundational Ideas

Bayes' theorem calculates a conditional probability from the reverse conditional probability and the relevant marginal probabilities.

A positive medical test is evidence of disease, but it does not establish a diagnosis. Bayes' theorem reasons from observed evidence back to a hypothesis, combining its prior probability with the likelihood of the evidence to obtain a posterior probability. section 2.1 section 10.2.1

P(H | D)= P(D | H) × P(H)P(D)

The theorem states P(H | D) = P(D | H) × P(H) / P(D), where H is a hypothesis and D is observed data. section 2.2.1

Each term has a distinct role:

  • P(H) is the prior probability of the hypothesis before the new evidence. section 2.2.1
  • P(D | H) is the likelihood: the probability of the observed data if the hypothesis is true. section 2.2.1
  • P(D) is the marginal probability of the data across competing hypotheses. It normalizes the result. section 2.2.1
  • P(H | D) is the posterior probability: the revised probability of the hypothesis after observing the data. section 2.2.1

A positive test for a rare disease

Suppose disease prevalence is 0.001, test sensitivity is 0.99, and the false-positive rate is 0.01. A positive result can occur with or without disease, so its overall probability is (0.99 × 0.001) + (0.01 × 0.999) = 0.01098. The posterior probability of disease after a positive result is (0.99 × 0.001) / 0.01098, or about 9%. The result is informative but not conclusive: the low base rate makes false positives important. section 2.2.2 section 10.2.1

As new evidence arrives, a posterior can become the prior for a further update. The result remains dependent on the prior and likelihood used; compelling evidence alone need not make a rare hypothesis likely when alternatives remain plausible. section 2.2 section 2.7

Bayesian neural networks apply the same structure to parameter uncertainty: training data update prior distributions over weights into posterior distributions. Predictions combine results across plausible parameter values, although exact posterior calculations are generally impractical for large networks. section 11.4.3

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines prior, likelihood, marginal probability, and posterior, then works through a numerical diagnostic calculation.
Examples
Illustrates the theorem with a rare-disease test and probability distributions over neural-network parameters.
What it assumes
Builds on the addition, multiplication, and complement rules of probability.
What you can do after
Enables readers to evaluate a positive test by calculating a posterior probability and recognizing its diagnostic limits.
How it connects
Connects conditional probability and the law of total probability to Bayesian inference and Bayesian neural networks.

Also explained in

Compare how 3 books explain Bayes' theorem →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.