A retrieved document can be relevant to a question without containing every fact needed for the answer. RAG-Token addresses this by allowing evidence from different retrieved documents to influence different generated tokens. Unlike a system that uses only the top-ranked passage, it can draw on lower-ranked evidence when that evidence better supports part of the response. section 4.3 section 4.3.2
The distinction is where each variant combines evidence from the top K retrieved documents:
- RAG-Sequence scores a complete answer against each document, then combines those answer-level scores. Each answer hypothesis stays with one document. section 4.3
- RAG-Token combines document-weighted predictions at each token position. Its document emphasis can therefore change as the answer develops; “switching documents” describes this behavior, not a requirement to make one hard selection per token. section 4.3.2 section 4.3.3
Comparing two quarterly reports
Suppose a financial analyst asks for a comparison of AWS and Azure cloud revenue growth. One retrieved report supplies the AWS figure; another supplies the Azure figure. Token-level evidence mixing can support the first part of the answer with the Amazon report and the next part with the Microsoft report, rather than requiring one report to carry the entire comparison. section 4.3 section 4.3.2
Listing 4.1 Where the sum over documents goes.
import numpy as np
def marginalize(p_z, p_tok):
"""p(y|x) under sequence-level and token-level marginalization."""
seq = np.sum(p_z * np.prod(p_tok, axis=1)) #A
tok = np.prod(np.sum(p_z[:, None] * p_tok, axis=0)) #B
return float(seq), float(tok)
p_z = np.array([0.55, 0.30, 0.15]) #C
split = np.array([[0.80, 0.10], # names the city, not the state
[0.10, 0.85], # names the state, not the city
[0.20, 0.20]])
whole = np.array([[0.80, 0.75], # one passage carries both
[0.10, 0.10],
[0.20, 0.20]])
for label, p_tok in [("evidence split", split),
("one passage has it all", whole)]:
seq, tok = marginalize(p_z, p_tok)
print(f"{label:22s} sequence={seq:.4f} token={tok:.4f}")The short NumPy example shows the defining operation: RAG-Sequence sums over documents after multiplying token probabilities; RAG-Token sums over documents for each token before multiplying. When two passages each support a different token, the example gives the answer a higher token-level score. section 4.3.3
Finer-grained mixing is not an automatic quality win. The reported open-domain question-answering results are close, with RAG-Sequence ahead on three of four datasets; RAG-Token’s clearer advantage appears when an output genuinely needs several documents. Both approaches have decoding costs: RAG-Token combines evidence throughout generation, while RAG-Sequence runs a separate beam search per document. Choose based on the answer’s need for cross-document synthesis, not on a simple claim that one variant is always cheaper or more accurate. section 4.3.3
Table 4.1 RAG sequence and RAG-token trade-offs
|
Model variant
|
NQ
|
TriviaQA
|
WebQuestions
|
CuratedTrec
|
|
RAG-Token
|
44.1
|
55.2
|
45.5
|
50.0
|
|
RAG-Sequence
|
44.5
|
56.8
|
45.2
|
52.2
|
The benchmark table puts the selection trade-off in context: token-level mixing does not consistently beat sequence-level mixing on open-domain question answering. section 4.3.3
