liveBook · Manning

Bayesian inference

As explained in Timeless Algorithms, The Foundational Ideas

Bayesian inference updates prior probabilities or distributions about hypotheses or parameters with observed data.

A positive test result is evidence of disease, but it is not the same as a diagnosis. Bayesian inference reasons backward from observed evidence to the probabilities of competing hypotheses. It combines what was plausible before the observation with how well each hypothesis explains it, producing revised probabilities that can be updated again as evidence accumulates. section 2.1 section 10.2.1

For a hypothesis H and evidence D, Bayes’ theorem states: P(H | D) = P(D | H) × P(H) / P(D). The prior P(H) is the probability before observing D; the likelihood P(D | H) measures how probable the evidence is if H holds; and the posterior P(H | D) is the revised probability. The denominator P(D) accounts for the evidence under all competing hypotheses. section 2.1 section 2.2.2

A positive test for a rare disease

Suppose disease prevalence is 0.1%, the test detects 99% of cases, and its false-positive rate is 1%. The probability of a positive result is 0.99 × 0.001 + 0.01 × 0.999 = 0.01098. The probability of disease given a positive result is therefore (0.99 × 0.001) / 0.01098, or about 9%. The result raises concern but is not conclusive: because the disease is rare, false positives outnumber true positives. section 2.2.2 section 10.2.1

The same logic applies to an unknown parameter θ. Its posterior distribution is p(θ | x) = p(x | θ)p(θ) / ∫p(x | θ)p(θ)dθ, where x is the observed data. The likelihood times the prior determines the posterior’s shape; the integral normalizes it to a probability distribution. That integral may have no closed-form solution. section 7.4.1

A conjugate prior simplifies repeated updating: combining it with a likelihood gives a posterior in the same distribution family. For example, a Beta prior and Binomial likelihood produce a Beta posterior, so observations change the distribution’s parameters rather than its family. When exact solutions are impractical, approximate methods such as Markov Chain Monte Carlo estimate the posterior through repeated sampling. section 7.4.1 section 2.5

Posterior probabilities can inform forecasts or be combined with the expected value of actions in decision analysis. They do not eliminate uncertainty or guarantee sound decisions: a strong prior can overwhelm evidence, while biased or unrepresentative data can distort the posterior. Nor should P(D | H) be confused with P(H | D); reversing these conditional probabilities can lead to serious errors. section 2.5 section 2.6

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Derives posterior normalization, explains conjugate updating, and introduces sampling when exact solutions are impractical.
Examples
Works through a rare-disease test and an offender-matching algorithm to illustrate base rates and false positives.
What you can do after
Evaluate and understand posterior probabilities by combining prior information with evidence and recognizing common interpretation errors.
How it connects
Connects conditional probability, conjugate priors, Markov Chain Monte Carlo, and Bayesian decision analysis.

Also explained in

Compare how 11 books explain Bayesian inference →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.