liveBook · Manning

Chapter 2: From effect to cause: Thomas Bayes and the first algorithm of learning

What you will learn in chapter 2 of Timeless Algorithms, The Foundational Ideas

The introduction presents Bayesian reasoning as a way to infer unseen causes from observations and revise beliefs as evidence accumulates, grounded in the rules of probability.

After this chapter you will be able to

Section by section

Introduction

The introduction presents Bayesian reasoning as a way to infer unseen causes from observations and revise beliefs as evidence accumulates, grounded in the rules of probability.

Key terms: inverse probability, observed outcomes, underlying cause, Bayes’ Theorem, conditional probability, initial belief, new evidence, addition rule, complement rule, multiplication rule

2.1 From observation to belief: how Bayesian reasoning begins

This section introduces Bayesian reasoning as updating beliefs about unknown causes from observed evidence, illustrates it with Bayes’ thought experiment, and distinguishes probability from odds.

Key terms: Bayesian inference, inverse probability, prior beliefs, likelihood, posterior probabilities, geometric thought experiment, unknown probability, binary outcomes, probability, odds

2.2 From prior to posterior: how Bayes’ Theorem works

Introduces Bayes’ Theorem as a way to revise beliefs about competing explanations as evidence accumulates.

Key terms: Bayes' Theorem, prior beliefs, newly observed evidence, updated beliefs

2.2.1 The mathematical structure of Bayes’ Theorem

This subsection explains the roles of the prior, likelihood, evidence, and posterior in updating a hypothesis’s probability after observing data.

Key terms: prior probability, likelihood, posterior probability, hypothesis, data, marginal probability, evidence, normalizing constant, competing hypotheses

2.2.2 Bayesian inference in real-world scenarios

The section applies Bayesian inference to a diagnostic test and a surveillance match, showing why positive results must be interpreted in light of rare underlying events.

Key terms: prior probability, likelihood, sensitivity, true positive rate, false positive rate, law of total probability, posterior probability, positive test result, positive match, base rate fallacy

2.2.3 The structure that makes inference possible

This section explains that Bayes’ Theorem depends on foundational probability rules rather than standing apart from them.

Key terms: Bayes' Theorem, addition rule, multiplication rule, complement rule

2.3 Building the machinery: the rules that power the theorem

This section explains why the probability rules behind Bayes’ Theorem matter and distinguishes Bayes’ contribution from the invention of those rules.

Key terms: Bayes' Theorem, addition rule, multiplication rule, complement rule, Bayes, probability rules, learning from evidence

2.3.1 Partitioning belief: the addition rule

Explains when probabilities of alternative outcomes can be added, when they form a complete distribution, and how to avoid common misuses of the addition rule.

Key terms: addition rule, mutually exclusive, disjoint, collectively exhaustive, sample space, probability distribution, mutual exclusivity, independence, inclusion-exclusion principle, overlap

2.3.2 Contrasting outcomes: the complement rule

This section explains why an event and its opposite have probabilities that sum to one, and uses that rule to calculate an “at least one” probability efficiently.

Key terms: complement rule, complement, mutually exclusive, collectively exhaustive, at least one 4, neither die, independent

2.3.3 Expressing likelihoods: odds and expected value

This section explains how odds express the relative strength of beliefs and how expected value combines probabilities and consequences to compare decisions under uncertainty.

Key terms: odds, complement, likelihood ratio, prior odds, posterior odds, expected value, long-run average, outcomes, probabilities, decision trees

2.3.4 Linking outcomes: the logic of the multiplication rule

This section explains how the multiplication rule combines probabilities for joint events, how dependence changes the calculation, and how the same reasoning applies to sequences in predictive models.

Key terms: multiplication rule, joint probability, conditional probability, dependent events, independent events, with replacement, without replacement, sample space, repeated independent trials, fixed probabilities

2.3.5 Independent repetition: compound events

This section applies the multiplication rule to independent repeated trials, distinguishes an exact ordered sequence from a count in any order, and connects repeated observations to inference.

Key terms: independent repetition, independent and identically distributed (i.i.d.), multiplication rule, ordered sequence, permutations, combinations, successes and failures, likelihood-based inference, competing probability models, prior beliefs

2.3.6 Likelihood from repetition: binomial probability

This section explains how the binomial model turns repeated success-or-failure trials into probabilities for outcome counts and likelihoods for comparing assumed success rates.

Key terms: binomial model, trials, success, failure, independent and identically distributed, 𝑛, binomial probability, multiplication rule, combinatorics, binomial coefficient

2.4 Applications in machine learning and AI

This section connects Bayesian inference and basic probability rules to modern machine learning and AI, emphasizing that older ideas remain useful at scale.

Key terms: Bayesian updating, addition rule, multiplication rule, Bayes' Theorem, probabilistic reasoning, timeless probabilistic ideas

2.4.1 Bayesian inference in modern algorithms

This section shows how Bayesian updating supports classification, reasoning about connected variables, sequential choices, and experiments, while distinguishing BIC as a Bayesian-inspired model-selection tool.

Key terms: Naive Bayes, prior probability, likelihood, posterior probability, conditionally independent, joint likelihood, Bayesian networks, probabilistic graphical models, conditional probabilities, addition rule

2.4.2 The addition and multiplication probability rules at work

The section shows how the addition rule makes predictions across competing outcomes coherent and how the multiplication rule gives probabilities to sequences in modern learning models.

Key terms: addition rule, mutually exclusive, collectively exhaustive, probability distribution, softmax, multiplication rule, conditional probabilities, likelihood, sequence, transitions

2.4.3 The enduring legacy of Bayes

This section explains how Bayes’ original insight has endured through later methods and extends beyond machine learning and AI.

Key terms: Bayesian reasoning, beliefs, new evidence, reasoning under uncertainty, uncertainty, decision-making

2.5 Bayesian reasoning beyond ML and AI

This section shows how Bayesian reasoning supports forecasting and decisions beyond ML and AI, while noting computational and reliability limits.

Key terms: Bayesian inference, forecasting, prior knowledge, new evidence, uncertainty, Bayesian decision analysis, posterior probabilities, expected value, expected benefit, posterior distribution

2.6 Where Bayesian inference goes wrong

This section explains five ways Bayesian conclusions can go wrong because of flawed assumptions, evidence, or interpretation, even when Bayes’ Theorem is applied correctly.

Key terms: prior distribution, regularization, biased or unrepresentative evidence, posterior distribution, selection bias, measurement error, prosecutor's fallacy, conditional probability, prior probabilities, competing explanations

2.7 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 1: Seeing inside the black box Chapter 3: The algorithm of estimation: Ronald Fisher and maximum likelihood →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.