liveBook · Manning

Marginalization

As explained in Retrieval Augmented Generation, The Foundational Ideas

Marginalization is the mathematical operation of summing or integrating over unobserved alternatives to obtain a combined probability or result.

A retriever’s first result is not always the best evidence for an answer. In retrieval-augmented generation (RAG), marginalization treats the retrieved document as a hidden choice rather than committing to the top-ranked one. For each of the top-K documents, it combines the retriever’s probability for that document with the generator’s probability of producing an answer from it, then sums those contributions. A lower-ranked document can therefore matter more when it supports the answer much better. section 4.2

p(y|x) = ∑z ∈ Z  p(z|x) ⋅ (p(y|x,z)

The equation sums over retrieved documents: document relevance weights answer likelihood, and the weighted contributions give the answer’s probability. section 4.2

When the third result has the answer

Asked about the main cause of the 2008 financial crisis, a retriever ranks an article about Lehman Brothers above an analysis of subprime mortgages. The first article is relevant but focuses on a consequence. After the retrieval scores are converted to weights that sum to one, the subprime analysis receives a lower retrieval weight (0.199 versus 0.227) but much stronger support for the proposed answer (0.92 versus 0.15). Their products are 0.183 and 0.034, respectively. The example shows why retrieval rank alone should not decide which evidence drives an answer. section 4.2.1

Where the sum sits changes what the generator can use. RAG-Sequence sums over documents after scoring a complete answer against each one; RAG-Token sums over documents at each generated token. Token-level mixing helps when different passages support different parts of an answer. Sequence-level scoring can do better when one passage supports the whole answer, and the variants have different decoding costs. section 4.3 section 4.3.3

Listing 4.1 Where the sum over documents goes.
import numpy as np
def marginalize(p_z, p_tok):
"""p(y|x) under sequence-level and token-level marginalization."""
seq = np.sum(p_z * np.prod(p_tok, axis=1))  #A
tok = np.prod(np.sum(p_z[:, None] * p_tok, axis=0))  #B
return float(seq), float(tok)
p_z = np.array([0.55, 0.30, 0.15])  #C
split = np.array([[0.80, 0.10],   # names the city, not the state
[0.10, 0.85],   # names the state, not the city
[0.20, 0.20]])
whole = np.array([[0.80, 0.75],   # one passage carries both
[0.10, 0.10],
[0.20, 0.20]])
for label, p_tok in [("evidence split", split),
("one passage has it all", whole)]:
seq, tok = marginalize(p_z, p_tok)
print(f"{label:22s} sequence={seq:.4f} token={tok:.4f}")

The NumPy example places the document sum outside or inside the product over token probabilities, making the sequence-level and token-level distinction concrete. section 4.3.3

Explicit marginalization is part of canonical RAG, not a requirement for every practical RAG system. Naive RAG instead puts retrieved chunks into an LLM prompt and relies on the model to use them, avoiding joint training and the explicit probability calculation. Marginalization offers a principled way to account for uncertain retrieval, but its added machinery should be weighed against that simpler deployment pattern. section 4.2.2 section 4.7

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Works through normalized document weights and answer scores, then compares sequence-level and token-level calculations and decoding tradeoffs.
Examples
Uses a financial-crisis retrieval example and a NumPy comparison of evidence split across passages versus concentrated in one.
What it assumes
Introduces retrieved documents as latent variables and walks through probability weights; uses NumPy for the executable variant comparison.
What you can do after
Evaluate/understand when weighted multi-document evidence or token-level mixing could help, and when simpler prompt-based RAG avoids their complexity.
How it connects
Links latent variables and probability weighting to retrieval-augmented generation, RAG-Sequence, RAG-Token, and naive RAG.

Also explained in

Compare how 2 books explain marginalization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.