Marginal likelihood
Marginal likelihood, or Bayesian evidence, is the probability or density of observed data under a model, averaged over its unknown parameters or latent variables according to their prior distributions.
Retrieval Augmented Generation, The Foundational Ideas
Marginal likelihood is the probability of an observed outcome after summing or averaging over possible latent variables or retrieved documents.
- How it is used
- Links document retrieval to masked-word prediction by treating the retrieved document as a latent variable in REALM’s training objective.
- How deep it goes
- Works through probability terms, log-sum-exp code, and a numerical loss example; stops short of runnable end-to-end REALM pretraining.
- Examples
- Uses a three-document NumPy calculation and a rare-fact masked-word prediction to show how retrieved evidence changes probabilities.
- What you can do after
- Evaluate/understand how retrieval probabilities affect prediction loss and why REALM trains its retriever for task utility.
Read how Retrieval Augmented Generation, The Foundational Ideas explains marginal likelihood →
Grokking Bayes
The probability of observed data under a model after integrating or summing over the model's uncertain parameters according to their prior distribution.
- How it is used
- Compares how likely observed data are under competing models by averaging over each model’s hypotheses.
- How deep it goes
- Defines prior-weighted averaging, distinguishes discrete sums from continuous integration, and works through a beta-binomial implementation.
- Examples
- Uses tea-preference counts and a 500-trial conversion-rate comparison, alongside a SciPy calculation.
- What you can do after
- Apply it via SciPy’s comb and beta functions to compare beta-binomial models and calculate Bayes factors.
Bayesian Optimization in Action
A marginal likelihood is the probability or density of observed data after integrating over latent variables or model parameters.
- How it is used
- Uses data fit to select Gaussian-process mean and covariance hyperparameters.
- How deep it goes
- Defines likelihood-based fitting, demonstrates parameter comparison, and gives an optimization loop without deriving the probability calculation.
- Examples
- Compares constant means of 0 and 2 and tunes length and output scales on a three-point training dataset.
- What you can do after
- Apply it via GPyTorch and Adam to optimize Gaussian-process mean or covariance parameters.
Read how Bayesian Optimization in Action explains marginal likelihood →
Also covered in
What to know first
- closed formA closed-form expression is an exact finite formula built from standard mathematical operations and functions that gives a result directly rather than through iterative numerical approximation.
- hypothesisA hypothesis is a proposed proposition, explanation, prediction, or model whose validity or plausibility can be assessed against evidence.
- gradient descentGradient descent is an iterative optimization method that updates parameters in the direction opposite an objective function’s gradient to reduce the objective’s value.
Related concepts
- bayes factorproduces · A Bayes factor is the ratio of the marginal likelihoods of two competing models or hypotheses, quantifying their relative support from observed data.
- evidence lower boundused by · The evidence lower bound (ELBO) is a variational objective that lower-bounds a model’s log marginal likelihood and is optimized in variational inference.
- regularizationequivalent to · Regularization is a modeling technique that constrains learning or penalizes model complexity to discourage overfitting and improve generalization.
- likelihooduses · A likelihood is the probability of observed data under a model, viewed as a function of the model’s parameters or candidate hypotheses.
- marginal probabilityequivalent to · A marginal probability or distribution describes a variable or event without conditioning on other variables, obtained from a joint distribution by summing or integrating over them.
Questions about marginal likelihood
What is marginal likelihood?
Marginal likelihood, or Bayesian evidence, is the probability or density of observed data under a model, averaged over its unknown parameters or latent variables according to their prior distributions.
Which Manning books explain marginal likelihood?
Retrieval Augmented Generation, The Foundational Ideas, Grokking Bayes and Bayesian Optimization in Action.
What should I know before learning marginal likelihood?
Manning books that teach marginal likelihood usually introduce closed form, hypothesis and gradient descent first.
What is marginal likelihood also called?
Marginal likelihood is also called evidence.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


