liveBook · Manning

Marginal probability

As explained in Retrieval Augmented Generation, The Foundational Ideas

Marginal probability is the probability of an event obtained by summing joint probabilities over another variable.

Marginal probability sums the joint probabilities of all alternatives that produce the desired outcome. In REALM training, it combines probability across multiple documents that could support the answer, then applies a negative logarithm as the loss. section 3.3

Three-document retrieval

A query assigns probabilities 0.391, 0.233, and 0.376 to three documents. Combining their document probabilities with answer probabilities produces marginal values of 0.356 and 0.593, with a resulting loss of 0.7783.

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains the definition, negative-log loss, tensor operations, and numerical Python example, stopping short of runnable cluster training.
Examples
Demonstrates three candidate documents, log-sum-exp aggregation, loss calculation, and retrieval-score perturbations.
What it assumes
Assumes familiarity with candidate probabilities and retrieval scores while defining marginal log-likelihood through a concrete example.
What you can do after
Evaluate how retrieval-score changes affect marginal probability and the resulting training loss.
How it connects
Ties marginal probability to joint probability, log-likelihood, knowledge retrieval, dual encoders, and cross encoders.

Also explained in

Compare how 5 books explain marginal probability →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.