Suppose a test for a rare condition returns positive. Should you believe you have it? The test’s accuracy alone cannot answer that question. Bayesian statistics treats probability as a degree of confidence informed by evidence: a belief can change when an observation arrives. This differs from treating probability solely as the long-run frequency of an outcome in repeated trials, especially when the decision cannot be repeated. section 12.1 section 13.4
Start with a prior distribution over possible causes and a likelihood function describing how probable each observation is under each cause. Combine them into a joint distribution of cause–observation pairs. For a particular observation, filter to matching pairs and project back to causes to obtain a posterior distribution. The likelihood describes the relationship between causes and effects; the prior supplies how plausible the causes were before this observation. section 13.6 section 13.0
Why a positive result need not mean 99.5% confidence
Suppose one person in 30,000 has a condition. A cheap test is positive for 995 of every 1,000 people with it and negative for 995 of every 1,000 without it. After a positive result, the posterior weights are 199 for having the condition and 29,999 for not having it: the probability of having it is about 0.7%, not 99.5%. Although false positives are uncommon among unaffected people, that group is vastly larger. section 13.4 section 13.6
Evidence can arrive repeatedly. With prior weights of 999 for a normal coin and 1 for a double-headed coin, ten heads produce posterior weights of 999 and 1,024 respectively. Repeated observations can therefore overcome a strong prior, rather than making either the prior or the latest observation decisive by itself. section 13.4
For finite categorical distributions, prior, likelihood, and posterior can be represented as generic distribution types and manipulated in code. Continuous uncertainty calls for a different computational route: a posterior density can be formed from prior and likelihood weights, then sampled with the Metropolis algorithm. These are tools for evidence-based decisions, not a reason to ignore how the prior or likelihood was chosen. section 13.0 section 16.5 section 13.6
Listing 16.8 Completing the implementation of continuous posterior sampling
static class Extensions
{
public static Func<B, IContinuousDistribution> Posterior<B>(
this IContinuousDistribution prior,
Func<double, IWeightedDistribution<B>> likelihood,
double stddev = 1.0)
{
return (B evidence) =>
{
Func<double, double> posteriorPDF = a =>
prior.Weight(a) * likelihood(a).Weight(evidence);
Func<double, IContinuousDistribution> proposal =
current => Normal.Distribution(current, stddev);
return Metropolis.Distribution(posteriorPDF, prior, proposal);
};
}
}
The C# implementation makes the continuous update concrete: it weights each candidate by its prior and the likelihood of the observed evidence, then passes that posterior density to Metropolis for sampling. section 16.5
