Long-context RAG concatenates many retrieved documents into one large decoder-only model input. It is a practical prototyping default, but long prompts can be costly, confusing, and vulnerable to hallucination or “lost in the middle” failures. section 5.1
Its central trade-offs are:
- Self-attention cost grows quadratically with input length, making large collections expensive to compute. section 5.1
- Information near the beginning or end may receive more attention than evidence buried in the middle. section 5.1
- Fusion-in-Decoder encodes documents independently before generation, offering a more scalable alternative for very large document sets. section 5.1
Table 5.1 Classic FiD vs. big context
|
Feature
|
Classic FiD (Encoder-Decoder)
|
Long-Context RAG (Decoder-Only)
|
|
Architecture
|
T5 / ATLAS. Encodes each document
independently, then fuses them only during generation.
|
GPT / Llama. Concatenates all
documents into one giant sequence and processes them simultaneously.
|
|
Complexity
|
Linear. Encoding 100 documents
costs roughly 10 times the compute of encoding 10.
|
Quadratic. Self-attention costs
grow squarely with length. Encoding 100 documents is significantly more
expensive than encoding 10.
|
|
Lost in the Middle
|
Low Risk. Because the encoder
processes every document in isolation, it creates a distinct representation
for every passage before fusion occurs.
|
High Risk. Large Decoder-only
models often ignore information buried in the middle of a massive prompt
context.
|
|
Implementation
|
Harder. Requires fine-tuning and
hosting your own model (e.g., HuggingFace transformers).
|
Easier. Often just an API call or
a prompt template change.
|
|
Best For
|
Scale and Efficiency. When you
have a fixed domain (e.g., medical records) and need to synthesize answers
from 50–100+ documents cheaply and accurately.
|
Flexibility and Speed. When you
need to build a system today that handles general queries over a few dozen
documents without training.
|
Compare Long-context RAG with classic Fusion-in-Decoder across architecture, complexity, middle-context risk, and implementation. section 5.1
