liveBook · Manning

Marginal likelihood

As explained in Grokking Bayes

The probability of observed data under a model after integrating or summing over the model's uncertain parameters according to their prior distribution.

Suppose two models describe how many people prefer tea over coffee. Which model made the survey result more likely? A marginal likelihood answers that question by considering every tea-preference rate allowed by a model, not just one rate. It averages the likelihood of the observed data at each rate, weighted by the model’s prior probability for that rate. section 4.8 section 4.9

For a model M and observed data D, the marginal likelihood is p(D | M). If the model has possible parameter values θ, add up their prior-weighted likelihoods; for a continuous range of values, use an integral instead. A model earns a higher marginal likelihood when the data are likely under values it assigned substantial prior weight. section 4.9

This is also called *evidence*. In Bayes’ theorem, evidence is the denominator: it divides each prior × likelihood result so the posterior probabilities sum to 1. The distinction is what is being averaged over: a likelihood concerns one hypothesis, while a marginal likelihood accounts for the hypotheses within a model. section 2.2 section 4.9

Tea preferences

Let k of n surveyed people prefer tea. Represent the unknown preference rate with a Beta(α, β) distribution and the survey count with a binomial likelihood. Their marginal likelihood is C(n, k) × B(α + k, β + n − k) / B(α, β), where C is the binomial coefficient and B is the beta function. Here n is known; k is the observed count whose probability is being calculated. section 4.9 section 11.5

from scipy.special import comb, beta
 
def marginal_likelihood(k, n, a, b):
    return (
        comb(n, k)
        * beta(a + k, b + n - k)
        / beta(a, b)
    )

The SciPy function computes that tea-survey probability using comb for the binomial coefficient and beta for the beta-function terms. Change α and β, passed as a and b, to compare different prior models against the same observed count. section 4.9

To compare two models directly, divide their marginal likelihoods to obtain a Bayes factor. A value above 1 favors the model in the numerator; a value below 1 favors the other. The comparison reflects the whole distribution of rates each model considers, including rates that explain the data poorly. section 4.10 section 11.5

A conversion-rate comparison

In 500 trials with 63 conversions, compare a fixed conversion rate of 0.1 with a rate given a Beta(1, 1) distribution. The fixed-rate model uses the binomial likelihood; the distributed-rate model uses its marginal likelihood. Their Bayes factor is about 4.68 in favor of the fixed-rate model. When calculating very small likelihoods, a logarithmic scale helps avoid numerical inaccuracies. section 11.5

How Grokking Bayes covers it

How deep it goes
Defines prior-weighted averaging, distinguishes discrete sums from continuous integration, and works through a beta-binomial implementation.
Examples
Uses tea-preference counts and a 500-trial conversion-rate comparison, alongside a SciPy calculation.
What it assumes
Builds from prior probabilities and likelihoods before introducing prior-weighted averaging; the worked calculation also uses binomial counts.
What you can do after
Apply it via SciPy’s comb and beta functions to compare beta-binomial models and calculate Bayes factors.
How it connects
Connects Bayes’ theorem, priors, likelihoods, posterior normalization, beta-binomial models, and Bayes factors.

Also explained in

Compare how 3 books explain marginal likelihood →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Bayes. The section links point to the parts of the book it draws on.