liveBook · Manning

Marginal likelihood

As explained in Retrieval Augmented Generation, The Foundational Ideas

Marginal likelihood is the probability of an observed outcome after summing or averaging over possible latent variables or retrieved documents.

Suppose several retrieved passages could help predict a masked word. Choosing one passage as the only correct support would discard useful evidence from the others. Marginal likelihood instead measures the probability of the word after accounting for the possible passages. section 3.3

In REALM, the retrieved document z is a latent variable. For an input x and correct word y, the model sums p(z | x) × p(y | z, x) over documents z. The first term is the retriever’s probability of selecting a document; the second is the encoder’s probability of predicting the word when given that document. Training maximizes the log of this sum, linking retriever and encoder performance. section 3.2

What a better retrieval score changes

In a three-document NumPy calculation, increasing the score of a useful document raises its retrieval probability from 0.391 to 0.826. The resulting masked-word loss falls from 0.7783 to 0.4413. The retriever is rewarded because putting more weight on that document improves the prediction—not merely because its embedding looks similar to the query. section 3.3

# [batch_size, num_candidates]
candidate_score = retrieval_score
candidate_log_probs = tf.math.log_softmax(candidate_score)
…
# [batch_size, num_candidates, num_masks]
joint_gold_log_probs = (
    tf.expand_dims(candidate_log_probs, -1) + gold_log_probs
)
# [batch_size, num_masks]
marginal_gold_log_probs = tf.reduce_logsumexp(joint_gold_log_probs, 1)

The TensorFlow operations add each candidate’s retrieval log-probability to its correct-word log-probability, then use log-sum-exp to marginalize across candidates. section 3.3

Summing over an entire corpus is not computationally practical, so REALM uses the retriever’s top-k candidates. A relevant passage can lift the marginal probability of a rare answer even when that passage has a modest retrieval probability. The practical constraint is that this approximation depends on which documents make the candidate set. section 3.2 section 3.4

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Works through probability terms, log-sum-exp code, and a numerical loss example; stops short of runnable end-to-end REALM pretraining.
Examples
Uses a three-document NumPy calculation and a rare-fact masked-word prediction to show how retrieved evidence changes probabilities.
What it assumes
Starts with a three-document probability example and explains the code’s score and loss calculations without requiring REALM pretraining experience.
What you can do after
Evaluate/understand how retrieval probabilities affect prediction loss and why REALM trains its retriever for task utility.
How it connects
Connects masked language modeling, dense retrieval, latent variables, cross-encoders, and top-k approximation.

Also explained in

Compare how 3 books explain marginal likelihood →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.