Chapter 2: From effect to cause: Thomas Bayes and the first algorithm of learning
The introduction presents Bayesian reasoning as a way to infer unseen causes from observations and revise beliefs as evidence accumulates, grounded in the rules of probability.
After this chapter you will be able to
- Reasoning from effects to causes. Distinguish Bayes’ question of inferring an unseen cause from an observed outcome from predicting an outcome under known conditions.
- Evidence-driven belief revision. Explain that Bayes’ Theorem gives a formal way to revise an initial probability for a hypothesis as new evidence accumulates.
- Inference from observed evidence. Explain how Bayesian inference reasons from observed evidence toward an unknown explanation, using prior beliefs and how well each hypothesis explains the evidence to update probabilities.
- Belief revision with evidence. Explain how Bayes’ Theorem combines prior beliefs with new evidence to update the plausibility of competing explanations, rather than starting over or treating a conclusion as final.
- Prior Likelihood and Posterior Roles. Explain how an initial belief about a hypothesis and the probability of the observed data under that hypothesis contribute to an updated belief after the data is observed.
- Positive Result Probability. Describe the quantities needed to interpret a positive result: the prior probability of the condition, the true-positive rate, the false-positive rate, and the complementary prior. Explain that the overall chance of a positive result includes both true positives and false positives.
- Base Rate Fallacy. Explain why rare conditions or events can yield mostly false positives despite high detection rates, and describe how Bayesian inference avoids overvaluing a positive result.
- Bayes’ foundational probability rules. Explain that Bayes’ Theorem updates probabilities using general rules of probability, especially the addition, multiplication, and complement rules, rather than replacing those rules.
- Probability rules behind Bayes’ Theorem. Explain why knowing Bayes’ Theorem’s equation is not enough for a complete understanding: the addition, multiplication, and complement rules provide its probability foundations.
- Addition of disjoint outcomes. Explain that the probability of any one of several mutually exclusive outcomes occurring is the sum of their individual probabilities, because the outcomes cannot occur together.
- Complete outcome distributions. Distinguish mutually exclusive outcomes from collectively exhaustive ones, and explain when their probabilities sum to 1. Apply the distinction to the die or prediction examples.
- Overlapping events and independence. Explain why directly adding probabilities fails for overlapping events and how the overlap is accounted for; distinguish mutual exclusivity from independence as a condition for this rule.
- Event and complement probabilities. Explain what an event’s complement contains and why its probability is one minus the event’s probability.
- At least one four calculation. Apply the complement rule to find the probability of at least one 4 when rolling two fair dice, and explain why calculating the opposite event is simpler.
- Odds as relative belief. Explain how odds compare the probability of an event with the probability that it does not occur, and how new evidence changes those odds.
- Expected value and repeated trials. Describe how expected value weights each possible outcome by its probability, and explain why it represents a long-run average rather than a guaranteed result of one or a few trials.
- Conditional multiplication for joint events. Explain how to find the probability that two events both occur when the first can affect the second. Apply the rule to an ace followed by a king drawn without replacement.
- Replacement and independence assumptions. Distinguish dependent from independent events and describe how replacing or not replacing a drawn card affects which form of the multiplication rule applies.
- Conditional chains in predictive models. Explain how the multiplication rule combines probabilities across stages in the section’s speech-recognition and Bayesian-network examples.
- Independent trials and sequence probability. Explain what independent, identically distributed trials require and how the multiplication rule gives the probability of one exact sequence of outcomes.
- Ordered sequences versus outcome counts. Distinguish the probability of one ordered arrangement from the probability of a given number of successes in any order, and describe why counting arrangements matters.
- Conditions for binomial modeling. State when the binomial model applies: a fixed number of trials, two possible outcomes per trial, and independent, identically distributed trials with a constant success probability.
- From sequences to success counts. Explain how the model finds the probability of exactly 𝑘 successes in 𝑛 trials regardless of order: the multiplication rule gives the probability of one sequence, and “𝑛 choose 𝑘” counts the sequences with that outcome count.
- Outcome counts as likelihood evidence. Explain how a binomial outcome count can be compared with an assumed success probability, and why accumulating observations can favor some probability models over others.
- Combined probabilistic foundations. Explain that modern algorithms often combine Bayesian updating with addition and multiplication rules: the rules help allocate probability across competing outcomes and accumulate evidence across dependent events.
- Naive Bayes classification. Explain how a Naive Bayes classifier uses observed features, class priors, and feature likelihoods to choose a class, and why its conditional-independence assumption simplifies the calculation.
- Bayesian network inference. Describe how a Bayesian network updates beliefs when evidence arrives in a system of related variables, and how limited dependencies make reasoning manageable.
- Bayesian optimization search. Trace how Bayesian optimization uses results from evaluated configurations to choose what to test next when evaluations are costly.
- Thompson sampling decisions. Explain how Thompson sampling updates beliefs about option rewards and uses samples from those beliefs to make successive choices.
- Bayesian A/B comparison. Describe how Bayesian A/B testing updates beliefs about alternatives and what its comparison tells decision-makers.
- Probabilities Across Competing Outcomes. Explain why predicted probabilities for mutually exclusive, collectively exhaustive outcomes must sum to 1, and illustrate this with classification or next-token prediction.
- Likelihoods Across Event Sequences. Explain how the multiplication rule combines conditional probabilities of successive events into the likelihood of a whole sequence, using examples from the section.
- Continuity of Bayesian reasoning. Explain how later methods built on Bayes’ idea of updating beliefs with evidence rather than replacing it, even as their applications and complexity changed.
- Forecasts That Evolve With Evidence. Explain how Bayesian forecasting starts with existing knowledge and revises, rather than discards, that understanding as new evidence arrives while acknowledging remaining uncertainty.
- Probabilities and Decision Consequences. Explain that Bayesian decision analysis considers both posterior probabilities and the expected value of competing actions to choose the greatest expected benefit under uncertainty.
- Reasonable and flexible priors. Explain why a prior should reflect credible knowledge without being so strong that it overwhelms new evidence or so vague that it provides little guidance when data is limited.
- Quality of observed evidence. Describe how biased or unrepresentative observations distort a posterior and why Bayes’ Theorem cannot repair that problem on its own.
- Reversed conditional probabilities. Distinguish the probability of evidence under a hypothesis from the probability of the hypothesis given that evidence, and explain the prosecutor’s fallacy.
- Inference versus causal proof. Explain why a high posterior probability supports an explanation’s consistency with evidence but does not, on its own, establish a causal relationship.
- Uncertainty after updating. Explain why a posterior still expresses uncertainty rather than absolute truth, and identify conditions that can create false precision.
Section by section
Introduction
The introduction presents Bayesian reasoning as a way to infer unseen causes from observations and revise beliefs as evidence accumulates, grounded in the rules of probability.
- Reasoning from effects to causes: Distinguish Bayes’ question of inferring an unseen cause from an observed outcome from predicting an outcome under known conditions.
Why it matters: This reversal explains the problem Bayesian inference was designed to address. - Evidence-driven belief revision: Explain that Bayes’ Theorem gives a formal way to revise an initial probability for a hypothesis as new evidence accumulates.
Why it matters: It frames probability as a way to learn from evidence rather than only a fixed measure of chance. - Probability rules as foundation: Explain why Bayes’ Theorem is not the whole Bayesian framework: its account of belief revision rests on the formal rules of probability.
Why it matters: Understanding Bayesian inference requires more than treating its theorem as an isolated formula.
Key terms: inverse probability, observed outcomes, underlying cause, Bayes’ Theorem, conditional probability, initial belief, new evidence, addition rule, complement rule, multiplication rule
2.1 From observation to belief: how Bayesian reasoning begins
This section introduces Bayesian reasoning as updating beliefs about unknown causes from observed evidence, illustrates it with Bayes’ thought experiment, and distinguishes probability from odds.
- Inference from observed evidence: Explain how Bayesian inference reasons from observed evidence toward an unknown explanation, using prior beliefs and how well each hypothesis explains the evidence to update probabilities.
Why it matters: It captures why the same observation can change the plausibility of competing explanations. - Hidden-ball inference example: Describe how Bayes’ table-and-ball thought experiment uses repeated observations to gain confidence about an unseen position and its associated unknown probability.
Why it matters: The example makes inference from indirect evidence concrete. - Probability versus odds: Distinguish probability from odds by identifying what favorable outcomes are compared with in each case, using the ace-drawing example.
Why it matters: The distinction prevents two ways of expressing uncertainty from being confused.
Key terms: Bayesian inference, inverse probability, prior beliefs, likelihood, posterior probabilities, geometric thought experiment, unknown probability, binary outcomes, probability, odds
2.2 From prior to posterior: how Bayes’ Theorem works
Introduces Bayes’ Theorem as a way to revise beliefs about competing explanations as evidence accumulates.
- Belief revision with evidence: Explain how Bayes’ Theorem combines prior beliefs with new evidence to update the plausibility of competing explanations, rather than starting over or treating a conclusion as final.
Why it matters: This gives you a way to reason under uncertainty without ignoring what you already knew or closing off future revision.
Key terms: Bayes' Theorem, prior beliefs, newly observed evidence, updated beliefs
2.2.1 The mathematical structure of Bayes’ Theorem
This subsection explains the roles of the prior, likelihood, evidence, and posterior in updating a hypothesis’s probability after observing data.
- Prior Likelihood and Posterior Roles: Explain how an initial belief about a hypothesis and the probability of the observed data under that hypothesis contribute to an updated belief after the data is observed.
Why it matters: These distinctions let you describe precisely what changes when a hypothesis is evaluated against new evidence. - Evidence as Normalizing Probability: Describe what the evidence term represents and why it is included when producing posterior probabilities.
Why it matters: It explains why updating a hypothesis requires more than considering only how well that hypothesis fits the data.
Key terms: prior probability, likelihood, posterior probability, hypothesis, data, marginal probability, evidence, normalizing constant, competing hypotheses
2.2.2 Bayesian inference in real-world scenarios
The section applies Bayesian inference to a diagnostic test and a surveillance match, showing why positive results must be interpreted in light of rare underlying events.
- Positive Result Probability: Describe the quantities needed to interpret a positive result: the prior probability of the condition, the true-positive rate, the false-positive rate, and the complementary prior. Explain that the overall chance of a positive result includes both true positives and false positives.
Why it matters: Counting both ways to get a positive result prevents test accuracy from being mistaken for the probability that a positive case is genuine. - Posterior Results In Both Examples: State the approximate probability that a positive case is genuine in each example, and distinguish those probabilities from the tests’ true-positive rates.
Why it matters: The worked results show how different a posterior probability can be from an impressive-looking detection rate. - Base Rate Fallacy: Explain why rare conditions or events can yield mostly false positives despite high detection rates, and describe how Bayesian inference avoids overvaluing a positive result.
Why it matters: Remembering the base rate helps keep striking new evidence in proportion to how uncommon the event is.
Key terms: prior probability, likelihood, sensitivity, true positive rate, false positive rate, law of total probability, posterior probability, positive test result, positive match, base rate fallacy
2.2.3 The structure that makes inference possible
This section explains that Bayes’ Theorem depends on foundational probability rules rather than standing apart from them.
- Bayes’ foundational probability rules: Explain that Bayes’ Theorem updates probabilities using general rules of probability, especially the addition, multiplication, and complement rules, rather than replacing those rules.
Why it matters: This connection sets up the probability rules used to understand why Bayesian updating works.
Key terms: Bayes' Theorem, addition rule, multiplication rule, complement rule
2.3 Building the machinery: the rules that power the theorem
This section explains why the probability rules behind Bayes’ Theorem matter and distinguishes Bayes’ contribution from the invention of those rules.
- Probability rules behind Bayes’ Theorem: Explain why knowing Bayes’ Theorem’s equation is not enough for a complete understanding: the addition, multiplication, and complement rules provide its probability foundations.
Why it matters: The rules help explain why the theorem works, rather than leaving it as an equation to memorize. - Bayes’ contribution to probability rules: Distinguish Bayes’ contribution from inventing the probability rules: he recognized that established rules could be combined into a coherent framework for learning from evidence.
Why it matters: This distinction locates Bayes’ insight in how the rules work together, not in their individual invention.
Key terms: Bayes' Theorem, addition rule, multiplication rule, complement rule, Bayes, probability rules, learning from evidence
2.3.1 Partitioning belief: the addition rule
Explains when probabilities of alternative outcomes can be added, when they form a complete distribution, and how to avoid common misuses of the addition rule.
- Addition of disjoint outcomes: Explain that the probability of any one of several mutually exclusive outcomes occurring is the sum of their individual probabilities, because the outcomes cannot occur together.
Why it matters: This is the basic rule for combining probabilities of competing outcomes. - Complete outcome distributions: Distinguish mutually exclusive outcomes from collectively exhaustive ones, and explain when their probabilities sum to 1. Apply the distinction to the die or prediction examples.
Why it matters: A set of predictions forms a complete probability distribution only when it includes every possible outcome. - Overlapping events and independence: Explain why directly adding probabilities fails for overlapping events and how the overlap is accounted for; distinguish mutual exclusivity from independence as a condition for this rule.
Why it matters: Checking for overlap prevents an inflated probability of either event.
Key terms: addition rule, mutually exclusive, disjoint, collectively exhaustive, sample space, probability distribution, mutual exclusivity, independence, inclusion-exclusion principle, overlap
2.3.2 Contrasting outcomes: the complement rule
This section explains why an event and its opposite have probabilities that sum to one, and uses that rule to calculate an “at least one” probability efficiently.
- Event and complement probabilities: Explain what an event’s complement contains and why its probability is one minus the event’s probability.
Why it matters: This gives you a way to find either probability from the other. - At least one four calculation: Apply the complement rule to find the probability of at least one 4 when rolling two fair dice, and explain why calculating the opposite event is simpler.
Why it matters: Finding the opposite event can avoid counting many separate ways the desired event might happen.
Key terms: complement rule, complement, mutually exclusive, collectively exhaustive, at least one 4, neither die, independent
2.3.3 Expressing likelihoods: odds and expected value
This section explains how odds express the relative strength of beliefs and how expected value combines probabilities and consequences to compare decisions under uncertainty.
- Odds as relative belief: Explain how odds compare the probability of an event with the probability that it does not occur, and how new evidence changes those odds.
Why it matters: Odds give you a way to describe how accumulating evidence changes the balance between explanations. - Expected value and repeated trials: Describe how expected value weights each possible outcome by its probability, and explain why it represents a long-run average rather than a guaranteed result of one or a few trials.
Why it matters: Expected value lets you compare uncertain payoffs without mistaking an average for a promise about the next outcome. - Backward evaluation in decision trees: Describe how a decision tree uses expected values to compare actions with uncertain outcomes.
Why it matters: Decision trees make the use of expected value visible when a choice has several possible consequences. - Expected value versus decision heuristics: Distinguish expected value from maximax, maximin, and minimax regret as ways to compare uncertain alternatives.
Why it matters: These rules can favor different choices because they pay attention to different parts of an uncertain outcome.
Key terms: odds, complement, likelihood ratio, prior odds, posterior odds, expected value, long-run average, outcomes, probabilities, decision trees
2.3.4 Linking outcomes: the logic of the multiplication rule
This section explains how the multiplication rule combines probabilities for joint events, how dependence changes the calculation, and how the same reasoning applies to sequences in predictive models.
- Conditional multiplication for joint events: Explain how to find the probability that two events both occur when the first can affect the second. Apply the rule to an ace followed by a king drawn without replacement.
Why it matters: The conditional form lets you calculate joint probabilities without overlooking how an earlier outcome changes a later one. - Replacement and independence assumptions: Distinguish dependent from independent events and describe how replacing or not replacing a drawn card affects which form of the multiplication rule applies.
Why it matters: Checking whether probabilities change between events prevents an unjustified independence assumption. - Repeated trials and outcome arrangements: Describe how the section extends multiplication across repeated independent trials with fixed success and failure probabilities, and state the role of the binomial coefficient.
Why it matters: Repeated trials require accounting for both their probabilities and the different orders in which outcomes can occur. - Conditional chains in predictive models: Explain how the multiplication rule combines probabilities across stages in the section’s speech-recognition and Bayesian-network examples.
Why it matters: The rule connects a large sequence probability to smaller probabilities that reflect the model’s dependencies.
Key terms: multiplication rule, joint probability, conditional probability, dependent events, independent events, with replacement, without replacement, sample space, repeated independent trials, fixed probabilities
2.3.5 Independent repetition: compound events
This section applies the multiplication rule to independent repeated trials, distinguishes an exact ordered sequence from a count in any order, and connects repeated observations to inference.
- Independent trials and sequence probability: Explain what independent, identically distributed trials require and how the multiplication rule gives the probability of one exact sequence of outcomes.
Why it matters: This is the starting point for calculating probabilities from repeated observations. - Ordered sequences versus outcome counts: Distinguish the probability of one ordered arrangement from the probability of a given number of successes in any order, and describe why counting arrangements matters.
Why it matters: Whether order matters determines which probability question you are answering. - Repeated observations and model plausibility: Explain how accumulating observations can provide evidence for updating beliefs about the process that generated them.
Why it matters: Repeated trials can help reveal an underlying process, not merely record its outcomes.
Key terms: independent repetition, independent and identically distributed (i.i.d.), multiplication rule, ordered sequence, permutations, combinations, successes and failures, likelihood-based inference, competing probability models, prior beliefs
2.3.6 Likelihood from repetition: binomial probability
This section explains how the binomial model turns repeated success-or-failure trials into probabilities for outcome counts and likelihoods for comparing assumed success rates.
- Conditions for binomial modeling: State when the binomial model applies: a fixed number of trials, two possible outcomes per trial, and independent, identically distributed trials with a constant success probability.
Why it matters: These conditions tell you whether a count of successes can be assessed with this model. - From sequences to success counts: Explain how the model finds the probability of exactly 𝑘 successes in 𝑛 trials regardless of order: the multiplication rule gives the probability of one sequence, and “𝑛 choose 𝑘” counts the sequences with that outcome count.
Why it matters: Counting all qualifying orders is what turns a sequence probability into a probability for the observed total. - Outcome counts as likelihood evidence: Explain how a binomial outcome count can be compared with an assumed success probability, and why accumulating observations can favor some probability models over others.
Why it matters: This connects a probability calculation for repeated trials to learning from observed data. - Distribution around expected successes: Describe the section’s stated pattern as the number of trials increases: the binomial distribution becomes smoother and concentrates more tightly around the expected count, 𝑛𝑝, helping compare observations with theoretical expectations.
Why it matters: The expected success count gives a reference point for comparing data with the model.
Key terms: binomial model, trials, success, failure, independent and identically distributed, 𝑛, binomial probability, multiplication rule, combinatorics, binomial coefficient
2.4 Applications in machine learning and AI
This section connects Bayesian inference and basic probability rules to modern machine learning and AI, emphasizing that older ideas remain useful at scale.
- Combined probabilistic foundations: Explain that modern algorithms often combine Bayesian updating with addition and multiplication rules: the rules help allocate probability across competing outcomes and accumulate evidence across dependent events.
Why it matters: It helps explain why several foundational probability ideas appear together in learning systems. - Timeless ideas at scale: Distinguish algorithms that explicitly use Bayes’ Theorem from those shaped by addition and multiplication rules without making Bayesian inference their primary focus. State the section’s broader claim that sophisticated learning methods often apply longstanding probabilistic ideas at unprecedented scale.
Why it matters: It prevents treating every use of foundational probability in AI as an explicit use of Bayes’ Theorem.
Key terms: Bayesian updating, addition rule, multiplication rule, Bayes' Theorem, probabilistic reasoning, timeless probabilistic ideas
2.4.1 Bayesian inference in modern algorithms
This section shows how Bayesian updating supports classification, reasoning about connected variables, sequential choices, and experiments, while distinguishing BIC as a Bayesian-inspired model-selection tool.
- Naive Bayes classification: Explain how a Naive Bayes classifier uses observed features, class priors, and feature likelihoods to choose a class, and why its conditional-independence assumption simplifies the calculation.
Why it matters: This explains why a simple probabilistic model can classify items such as emails efficiently. - Bayesian network inference: Describe how a Bayesian network updates beliefs when evidence arrives in a system of related variables, and how limited dependencies make reasoning manageable.
Why it matters: It shows how Bayesian updating can work across connected variables rather than a single classification decision. - Bayesian optimization search: Trace how Bayesian optimization uses results from evaluated configurations to choose what to test next when evaluations are costly.
Why it matters: It explains how to learn from expensive trials instead of testing every configuration. - Thompson sampling decisions: Explain how Thompson sampling updates beliefs about option rewards and uses samples from those beliefs to make successive choices.
Why it matters: It illustrates a concrete way to act while still learning which option works best. - Bayesian A/B comparison: Describe how Bayesian A/B testing updates beliefs about alternatives and what its comparison tells decision-makers.
Why it matters: It connects uncertain experimental results to a comparison a decision-maker can use. - BIC model selection boundary: Distinguish BIC from the direct Bayesian updating methods here and explain how it ranks competing models.
Why it matters: It prevents a Bayesian-inspired model score from being mistaken for a full Bayesian inference procedure.
Key terms: Naive Bayes, prior probability, likelihood, posterior probability, conditionally independent, joint likelihood, Bayesian networks, probabilistic graphical models, conditional probabilities, addition rule
2.4.2 The addition and multiplication probability rules at work
The section shows how the addition rule makes predictions across competing outcomes coherent and how the multiplication rule gives probabilities to sequences in modern learning models.
- Probabilities Across Competing Outcomes: Explain why predicted probabilities for mutually exclusive, collectively exhaustive outcomes must sum to 1, and illustrate this with classification or next-token prediction.
Why it matters: This explains why a model can report probabilities for alternatives, not just a winning label or score. - Likelihoods Across Event Sequences: Explain how the multiplication rule combines conditional probabilities of successive events into the likelihood of a whole sequence, using examples from the section.
Why it matters: It explains how models assign a likelihood to a path, observation series, or piece of text rather than only to one step.
Key terms: addition rule, mutually exclusive, collectively exhaustive, probability distribution, softmax, multiplication rule, conditional probabilities, likelihood, sequence, transitions
2.4.3 The enduring legacy of Bayes
This section explains how Bayes’ original insight has endured through later methods and extends beyond machine learning and AI.
- Continuity of Bayesian reasoning: Explain how later methods built on Bayes’ idea of updating beliefs with evidence rather than replacing it, even as their applications and complexity changed.
Why it matters: It shows why a foundational idea can remain useful even when the methods built on it look very different. - Bayesian reasoning beyond AI: Describe why the section presents Bayesian reasoning as relevant beyond ML and AI: decisions in many fields must account for uncertainty.
Why it matters: It places Bayesian reasoning in the broader setting of making judgments when outcomes are uncertain.
Key terms: Bayesian reasoning, beliefs, new evidence, reasoning under uncertainty, uncertainty, decision-making
2.5 Bayesian reasoning beyond ML and AI
This section shows how Bayesian reasoning supports forecasting and decisions beyond ML and AI, while noting computational and reliability limits.
- Forecasts That Evolve With Evidence: Explain how Bayesian forecasting starts with existing knowledge and revises, rather than discards, that understanding as new evidence arrives while acknowledging remaining uncertainty.
Why it matters: It explains how a forecast can improve with new information without pretending uncertainty has vanished. - Probabilities and Decision Consequences: Explain that Bayesian decision analysis considers both posterior probabilities and the expected value of competing actions to choose the greatest expected benefit under uncertainty.
Why it matters: The most likely outcome alone does not tell you which action has the greatest expected benefit. - Sampling for Complex Posteriors: Describe why complex Bayesian models may need approximate inference and how methods such as MCMC estimate a posterior through repeated sampling rather than direct calculation.
Why it matters: Approximation makes Bayesian inference usable when direct calculation is impractical. - Limits of Bayesian Reliability: Explain that Bayesian inference can support informed decisions under uncertainty, but its reliability depends on the assumptions, data, and judgments used.
Why it matters: A disciplined way to update beliefs still requires scrutiny of what goes into it.
Key terms: Bayesian inference, forecasting, prior knowledge, new evidence, uncertainty, Bayesian decision analysis, posterior probabilities, expected value, expected benefit, posterior distribution
2.6 Where Bayesian inference goes wrong
This section explains five ways Bayesian conclusions can go wrong because of flawed assumptions, evidence, or interpretation, even when Bayes’ Theorem is applied correctly.
- Reasonable and flexible priors: Explain why a prior should reflect credible knowledge without being so strong that it overwhelms new evidence or so vague that it provides little guidance when data is limited.
Why it matters: An unreasonable starting belief can distort the conclusion despite correct updating. - Quality of observed evidence: Describe how biased or unrepresentative observations distort a posterior and why Bayes’ Theorem cannot repair that problem on its own.
Why it matters: A disciplined calculation cannot make poor-quality observations representative. - Reversed conditional probabilities: Distinguish the probability of evidence under a hypothesis from the probability of the hypothesis given that evidence, and explain the prosecutor’s fallacy.
Why it matters: Reversing those probabilities can turn evidence into a misleading conclusion about a hypothesis. - Inference versus causal proof: Explain why a high posterior probability supports an explanation’s consistency with evidence but does not, on its own, establish a causal relationship.
Why it matters: Strong statistical support should not be mistaken for proof of what caused an outcome. - Uncertainty after updating: Explain why a posterior still expresses uncertainty rather than absolute truth, and identify conditions that can create false precision.
Why it matters: Acknowledging remaining uncertainty prevents an updated belief from being presented as certainty.
Key terms: prior distribution, regularization, biased or unrepresentative evidence, posterior distribution, selection bias, measurement error, prosecutor's fallacy, conditional probability, prior probabilities, competing explanations
2.7 Summary
Where this is used later
- Estimating unknown parameters (chapter 3) · The rare-disease example and Bayesian updating
- Revisiting Bayes: from diagnosis to decision (chapter 7) · Bayesian updating of disease probability after diagnostic evidence.
- From posterior probability to expected utility (chapter 7) · The diagnostic use of a test result to update the probability of disease.
- Why logistic regression requires maximum likelihood (chapter 3) · Logistic regression builds on the distinction between probabilities and odds.
- The logistic model (chapter 3) · The distinction between probability and odds.
- Decision trees: visualizing choice under uncertainty (chapter 7) · That section develops decision trees as a way to visualize choices under uncertainty.
- What is likelihood? (chapter 3) · The discussion of likelihood develops the idea of evaluating observations under a probability model.
- Naïve Bayes (chapter 3) · Revisits Naive Bayes as a modeling application.
Concepts explained in this chapter
- Bayesian inferenceBayesian inference updates prior probabilities or distributions about hypotheses or parameters with observed data.
- multiplication ruleThe multiplication rule is a probability rule for calculating the probability that multiple events occur together from joint or conditional probabilities.
- Bayes' theoremBayes' theorem calculates a conditional probability from the reverse conditional probability and the relevant marginal probabilities.
- addition ruleThe addition rule calculates the probability that at least one of multiple mutually exclusive events occurs by summing their probabilities.
- independenceAn independent event is an event whose probability is unaffected by the occurrence or outcome of another event.
- posterior probabilityA posterior is a probability or probability distribution updated after incorporating observed evidence.
- prior probabilityA prior distribution represents uncertainty about an unknown quantity before incorporating observed data.
- complement ruleThe complement rule states that the probability of an event not occurring is one minus the probability of that event occurring.
- binomial distributionThe binomial distribution gives the probabilities of numbers of successes in a fixed number of independent trials with a constant success probability.
- oddsOdds are the ratio of the number or probability of favorable outcomes to the number or probability of unfavorable outcomes.
- dependent eventA dependent event is an event whose probability is affected by the occurrence or outcome of another event.
- BICThe Bayesian information criterion is a likelihood-based model-selection score that penalizes model complexity, with a penalty that increases with sample size.
- Bayesian A/B testingBayesian A/B testing compares alternatives by updating probability distributions about their performance using observed data.
- Bayesian networkA Bayesian network is a probabilistic graphical model that represents variables and their conditional dependencies using a directed acyclic graph.
- Bayesian optimizationBayesian optimization is a method for optimizing an expensive or black-box objective by using a probabilistic surrogate model to guide evaluations.
- Thompson samplingThompson sampling is a Bayesian sequential decision algorithm that samples from posterior distributions over alternatives and selects the alternative with the highest sampled value.
- naive bayes classifierA naive Bayes classifier is a probabilistic classification model that applies Bayes' theorem while assuming features are conditionally independent given the class.
- evidenceEvidence is information or observations considered in support of a claim or conclusion.
- probabilityProbability is a numerical measure representing the likelihood of an event or outcome conditional on information and assumptions.
- meanExpected value is the probability-weighted average of the possible numerical outcomes of a random variable.
- independent and identically distributedIndependent and identically distributed describes random variables that are mutually independent and share the same probability distribution.
- likelihoodA likelihood function measures how compatible observed data are with different parameter values of a statistical model.
- base rate fallacyThe base rate fallacy is the error of neglecting an event's underlying prevalence when interpreting evidence about it.
- false-positive rateThe false positive rate is the proportion of actually negative cases incorrectly classified as positive.
- law of total probabilityThe law of total probability calculates an event's overall probability by summing its probabilities across mutually exclusive and collectively exhaustive cases.
- causationCausality, or causation, is a relationship in which a change in one factor brings about a change in another.
- prosecutor's fallacyThe prosecutor's fallacy is the error of confusing the probability of evidence given a hypothesis with the probability of the hypothesis given the evidence.
- independent repetitionIndependent repetition is the repeated performance of an experiment under identical conditions with independent outcomes across trials.
- sampling without replacementSampling without replacement is a sampling procedure in which each selected item is not returned to the sample space before the next trial.
- hyperparameterA hyperparameter is a setting that controls a model or learning procedure and is selected outside ordinary parameter fitting.
- decision treeA decision tree is a hierarchical predictive model that recursively partitions data using feature-based rules.
- maximax methodMaximax is a decision heuristic that selects the option with the highest possible outcome.
- maximin methodMaximin is a decision heuristic that selects the option with the best worst-case outcome.
- minimax regret methodMinimax regret is a decision heuristic that selects the option minimizing the worst possible regret relative to the best alternative in hindsight.
- exhaustive event setCollective exhaustiveness is the property of a set of outcomes that together they cover every possibility in a defined outcome space.
- mutually exclusive eventMutually exclusive events are events that cannot occur simultaneously.
- probabilistic reasoningProbabilistic reasoning is inference that represents and updates beliefs using probabilities under uncertainty.
- probability distributionA probability distribution assigns nonnegative probabilities to possible outcomes so that the probabilities over the defined outcome space sum to one.
- Markov chainA Markov chain is a stochastic model of transitions among states in which the next state depends only on the current state.
- reinforcement learningReinforcement learning is a machine-learning approach in which an agent learns behavior through interaction with an environment and feedback about its actions.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
