A retriever’s first result is not always the best evidence for an answer. In retrieval-augmented generation (RAG), marginalization treats the retrieved document as a hidden choice rather than committing to the top-ranked one. For each of the top-K documents, it combines the retriever’s probability for that document with the generator’s probability of producing an answer from it, then sums those contributions. A lower-ranked document can therefore matter more when it supports the answer much better. section 4.2
The equation sums over retrieved documents: document relevance weights answer likelihood, and the weighted contributions give the answer’s probability. section 4.2
When the third result has the answer
Asked about the main cause of the 2008 financial crisis, a retriever ranks an article about Lehman Brothers above an analysis of subprime mortgages. The first article is relevant but focuses on a consequence. After the retrieval scores are converted to weights that sum to one, the subprime analysis receives a lower retrieval weight (0.199 versus 0.227) but much stronger support for the proposed answer (0.92 versus 0.15). Their products are 0.183 and 0.034, respectively. The example shows why retrieval rank alone should not decide which evidence drives an answer. section 4.2.1
Where the sum sits changes what the generator can use. RAG-Sequence sums over documents after scoring a complete answer against each one; RAG-Token sums over documents at each generated token. Token-level mixing helps when different passages support different parts of an answer. Sequence-level scoring can do better when one passage supports the whole answer, and the variants have different decoding costs. section 4.3 section 4.3.3
Listing 4.1 Where the sum over documents goes.
import numpy as np
def marginalize(p_z, p_tok):
"""p(y|x) under sequence-level and token-level marginalization."""
seq = np.sum(p_z * np.prod(p_tok, axis=1)) #A
tok = np.prod(np.sum(p_z[:, None] * p_tok, axis=0)) #B
return float(seq), float(tok)
p_z = np.array([0.55, 0.30, 0.15]) #C
split = np.array([[0.80, 0.10], # names the city, not the state
[0.10, 0.85], # names the state, not the city
[0.20, 0.20]])
whole = np.array([[0.80, 0.75], # one passage carries both
[0.10, 0.10],
[0.20, 0.20]])
for label, p_tok in [("evidence split", split),
("one passage has it all", whole)]:
seq, tok = marginalize(p_z, p_tok)
print(f"{label:22s} sequence={seq:.4f} token={tok:.4f}")The NumPy example places the document sum outside or inside the product over token probabilities, making the sequence-level and token-level distinction concrete. section 4.3.3
Explicit marginalization is part of canonical RAG, not a requirement for every practical RAG system. Naive RAG instead puts retrieved chunks into an LLM prompt and relies on the model to use them, avoiding joint training and the explicit probability calculation. Marginalization offers a principled way to account for uncertain retrieval, but its added machinery should be weighed against that simpler deployment pattern. section 4.2.2 section 4.7

