Chapter 13: Conditional probability with Bayes’ theorem

Introduction

This chapter introduces using Bayes’ theorem with discrete probability distributions to update prior beliefs from observations and reason from effects back to likely causes.

• Bayesian posterior updates: Explain that Bayes’ theorem computes a posterior distribution by updating a prior opinion using an observation. Describe the chapter’s aim of using this update to infer how likely possible causes are after an effect is observed.
  Why it matters: Posterior updating is the central operation that turns probabilistic models into tools for reasoning from evidence.
• Cause-and-effect statistical models: Describe the chapter’s proposed use of statistical models that represent relationships between causes and effects. Explain that these models support computing whether a particular cause is correlated with an observed effect.
  Why it matters: A cause-and-effect model supplies the structure that makes evidence useful for Bayesian inference.

Key terms: Bayes’ theorem, prior distribution, posterior distribution, observation, conditional distribution, statistical models, causes, effects, correlated, likelihood functions

13.1 Bayes’ theorem

This section derives Bayes’ theorem from survey-table probabilities and shows how evidence updates a prior belief into a posterior diagnosis.

• Marginal table probabilities: Describe how the cold-and-sneeze marginal table supplies prior, conditional, and joint probabilities. State what P(C), P(S|C), P(C|S), and P(C&S) mean in the survey example.
  Why it matters: Reading the margins and cells correctly is what makes the later Bayesian calculations meaningful.
• Bayes theorem derivation: Explain that the same joint probability can be written as P(C) × P(S|C) and as P(S) × P(C|S), yielding P(S)P(C|S) = P(C)P(S|C). Describe this as the symmetrical relationship between prior and conditional probabilities for the two events.
  Why it matters: Bayes’ theorem connects evidence-to-cause reasoning with cause-to-evidence probabilities.
• Prior-to-posterior update: Explain how observing a sneeze changes the belief that a person has a cold from the prior P(C)=10% to the posterior P(C|S)=76.1%. Distinguish a prior as a belief before additional evidence from a posterior as a belief after observing it.
  Why it matters: This evidence-based update is the practical purpose of Bayesian reasoning in the section.
• Likelihood function stability: Describe the likelihood of sneezing as a function of whether a person has a cold: P(S|C)=86% versus P(S|not C)=3%. Explain the model’s hypothesis that these conditional likelihoods may remain stable even when seasonal populations change the priors P(C) and P(S).
  Why it matters: Separating stable cause-to-effect likelihoods from changing population rates supports a useful disease model.

Key terms: marginal table, prior probability, conditional probability, joint probability, P(C), P(S|C), Bayes’ theorem, P(S)P(C|S) = P(C)P(S|C), posterior probability, evidence

13.2 Likelihood functions and joint distributions

This section represents conditional probabilities as likelihood functions and combines them with priors to construct joint distributions, including with integer weights.

• Likelihood function representation: Explain how a likelihood function represents conditional probability by taking a condition such as Cold.Yes or Cold.No and returning a distribution for an observation such as sneezing. Describe the cold-and-sneezing example, in which sneezing is 86% likely with a cold and 3% likely without one.
  Why it matters: A likelihood function turns a statistical model of causes and effects into code that can be applied to different priors.
• Joint distribution construction: Describe how sampling first from a prior and then from a likelihood function produces tuples such as (Cold, Sneezed), and explain why their distribution is a joint distribution. Compute or explain that each joint probability is the prior probability of a condition multiplied by the conditional likelihood of the observation given that condition.
  Why it matters: The joint distribution captures the probabilities of causes and observations together, which is the basis for later Bayesian reasoning.
• Integer weight normalization: Explain why directly multiplying integer distribution weights can require a fractional division, and describe the LCM-based solution for constructing exact integer-weight joint distributions. State that the prior weight is multiplied by the least common multiple of all conditional-distribution total weights so division by the applicable conditional total remains integral.
  Why it matters: This preserves the book's integer-weight distribution representation while allowing a prior and likelihood function to be combined exactly.

Key terms: likelihood function, conditional probability, prior, Cold, Sneezed, joint distribution, categorical distribution, support, P(C & S), integer weights

13.3 Updating priors by reasoning from effects to causes

This section shows how to update a prior belief into a posterior distribution by observing an effect, constructing a joint distribution, and filtering it to reason back to possible causes.

• Posterior diagnostic reasoning: Explain how observing an effect, such as a sneeze, changes a prior belief about a possible cause, such as having a cold, into a posterior probability. State that diagnosis reasons backward from observed effects to their possible causes.
  Why it matters: Bayesian reasoning lets you revise beliefs from evidence instead of treating evidence as a direct proof of a cause.
• Joint-distribution posterior computation: Describe how to compute a posterior likelihood function from a prior and a likelihood function: form the joint distribution with ToJoint, then Filter and Project it to retain outcomes consistent with the observed evidence. Explain that filtering out inconsistent possibilities and projecting the remaining distribution produces a distribution over causes for each observed effect.
  Why it matters: This turns Bayes’ theorem into an operation on finite discrete distributions rather than requiring manual probability-table calculations.
• Bayes theorem code correspondence: Trace the correspondence between Bayes’ theorem and the distribution code: the prior is P(A), the likelihood function is P(B|A), ToJoint produces P(A&B), and conditioning through Filter and Project corresponds to dividing by P(B) to obtain P(A|B).
  Why it matters: Seeing the correspondence explains why the distribution operations correctly implement Bayesian conditioning.
• Sneezing posterior proportions: State the worked cold-and-sneeze posterior: among the 1,130 people who sneezed, 860 have a cold and 270 do not, giving posterior weights in the reduced ratio 86 to 27. Explain that this is the conditional distribution for cold status given a sneeze.
  Why it matters: The example verifies that filtering the joint distribution produces the same posterior proportions obtained from the marginal table.

Key terms: prior probability, posterior probability, conditional probability, diagnosis, P(C), P(C|S), IDiscreteDistribution<A>, Func<A, IDiscreteDistribution<B>>, ToJoint, Filter

13.4 Some applications of Bayesian reasoning

This section demonstrates Bayesian reasoning through medical testing, coin flips, and programmer-relevant inference applications, emphasizing how priors and likelihoods determine posteriors.

• Medical-test base-rate inference: Explain why a positive 99.5%-accurate test for Frob Syndrome yields only about a 0.7% probability of having the syndrome when its prior prevalence is 1 in 30,000.
  Why it matters: Test accuracy alone does not tell you what a positive result means for an individual.
• Joint-distribution explanation: Describe the population-count explanation behind the Frob result: among 6 million people, approximately 199 true positives are outweighed by 29,999 false positives.
  Why it matters: Viewing the joint distribution as counts makes an unintuitive posterior easier to check and communicate.
• Ten-heads coin posterior: Trace why drawing from 999 normal coins and one double-headed coin, then observing ten heads in a row, produces a posterior slightly above 50% for the double-headed coin.
  Why it matters: This example shows how strong evidence can overcome a very small prior when competing explanations make the observation similarly likely.
• Bayesian inference application pattern: Describe the reusable application pattern: derive a prior distribution and a likelihood function from data, then compute a posterior that identifies the most likely hidden cause or category given observations.
  Why it matters: This is the practical workflow that turns observed data into probabilistic decisions in software systems.
• Programmer-relevant Bayesian examples: State how the same prior-likelihood-posterior structure applies to examples such as spam classification, router fault diagnosis, moderator disagreement, welding safety, and IDE function completion.
  Why it matters: Recognizing the shared structure helps you spot opportunities for Bayesian inference in software and operational systems.

Key terms: prior, posterior, likelihood function, test sensitivity, test specificity, false positive, joint distribution, true positive, false negative, posterior distribution

13.5 Unconditional likelihood functions are independent

This section shows that independent random quantities can be combined through an unconditional likelihood function, then projected into useful derived distributions such as a D20 roll with advantage.

• Independent joint distributions: Explain how an unconditional likelihood function represents a random quantity independent of the prior, and how the same prior-and-likelihood mechanism produces their joint distribution.
  Why it matters: This lets you combine random events whether one depends on the other or not.
• Roll with advantage distribution: Describe how rolling two independent fair D20 dice, forming their 400-pair joint distribution, and projecting each pair to its larger value models rolling with advantage.
  Why it matters: The example demonstrates how a joint distribution and projection can model a game rule exactly rather than merely simulate it.
• Advantage outcome weighting: State how rolling with advantage changes relative outcome weights: on a normal D20 a 20 and a 1 are equally likely, but with advantage a 20 is 39 times as likely as a 1.
  Why it matters: This quantifies why keeping the higher of two rolls is an enormous advantage.
• Combined projection helper: Describe the helper function's purpose: combine two dependent or independent distributions into their joint distribution and immediately apply a projection such as sum, maximum, or minimum.
  Why it matters: The helper makes a recurring distribution-combination pattern concise while preserving its expressive power.

Key terms: unconditional likelihood function, independent random events, joint distribution, prior, likelihood function, roll with advantage, d20, discrete uniform distribution, joint probability, projection

13.6 Summary

This summary recaps how priors and likelihood functions produce posterior estimates through Bayesian reasoning, and why priors prevent misleading interpretations of evidence.