chapter two

2 From effect to cause: Thomas Bayes and the first algorithm of learning

 

This chapter covers

  • Thomas Bayes’ An Essay Towards Solving a Problem in the Doctrine of Chances (1763) and its geometric approach to inverse probability.
  • How Bayes’ Theorem updates the probabilities of competing hypotheses using new evidence
  • Why the foundational rules of probability are essential to Bayesian inference
  • How Bayesian reasoning and the rules of probability continue to shape modern statistics, machine learning, and AI
  • Why common misconceptions about Bayes’ Theorem lead to flawed interpretation and misuse

How can one reason backward, from observed outcomes to the likelihood of an underlying cause? And once an initial conclusion has been reached, how should that belief change as new evidence becomes available? Those were the two fundamental questions Thomas Bayes addressed in An Essay Towards Solving a Problem in the Doctrine of Chances (1763). At a time when probability focused primarily on predicting future outcomes under known conditions, Bayes asked a different question: given what we observe, how should we reason about what we cannot directly see?

2.1 From observation to belief: how Bayesian reasoning begins

2.2 From prior to posterior: how Bayes’ Theorem works

2.2.1 The mathematical structure of Bayes’ Theorem

2.2.2 Bayesian inference in real-world scenarios

2.2.3 The structure that makes inference possible

2.3 Building the machinery: the rules that power the theorem

2.3.1 Partitioning belief: the addition rule

2.3.2 Contrasting outcomes: the complement rule

2.3.3 Expressing likelihoods: odds and expected value

2.3.4 Linking outcomes: the logic of the multiplication rule

2.3.5 Independent repetition: compound events

2.3.6 Likelihood from repetition: binomial probability

2.4 Applications in machine learning and AI

2.4.1 Bayesian inference in modern algorithms

2.4.2 The addition and multiplication probability rules at work

2.4.3 The enduring legacy of Bayes

2.5 Bayesian reasoning beyond ML and AI

2.6 Where Bayesian inference goes wrong

2.7 Summary