Lost in the middle describes a language-model failure in which relevant information buried inside a long context receives less attention than information near the beginning or end. Irrelevant retrieved material can crowd out or obscure the useful passage, degrading answer quality. section 9.1.1 section 5.1
The effect appears in two common architecture patterns:
- Passive retrieval adds documents for every prompt, including prompts that need no retrieval; irrelevant context consumes the budget and buries relevant evidence. section 9.1.1
- Concatenating many documents into one sequence makes the model process a confused, expensive context; Transformer self-attention costs grow quadratically with sequence length. section 5.1
- Fusion-in-Decoder reduces the risk by encoding documents independently and delaying fusion until generation, preserving distinct representations before synthesis. section 5.1
A mashed-together document set
Imagine ten different chapters mashed into one sequence, like ten half-shredded books glued together. Evidence at the beginning or end may receive more weight, while a relevant passage in the middle is overlooked. With 1,000 documents, prompt stuffing also becomes prohibitively slow and expensive. section 5.1
Table 5.1 Classic FiD vs. big context
|
Feature
|
Classic FiD (Encoder-Decoder)
|
Long-Context RAG (Decoder-Only)
|
|
Architecture
|
T5 / ATLAS. Encodes each document
independently, then fuses them only during generation.
|
GPT / Llama. Concatenates all
documents into one giant sequence and processes them simultaneously.
|
|
Complexity
|
Linear. Encoding 100 documents
costs roughly 10 times the compute of encoding 10.
|
Quadratic. Self-attention costs
grow squarely with length. Encoding 100 documents is significantly more
expensive than encoding 10.
|
|
Lost in the Middle
|
Low Risk. Because the encoder
processes every document in isolation, it creates a distinct representation
for every passage before fusion occurs.
|
High Risk. Large Decoder-only
models often ignore information buried in the middle of a massive prompt
context.
|
|
Implementation
|
Harder. Requires fine-tuning and
hosting your own model (e.g., HuggingFace transformers).
|
Easier. Often just an API call or
a prompt template change.
|
|
Best For
|
Scale and Efficiency. When you
have a fixed domain (e.g., medical records) and need to synthesize answers
from 50–100+ documents cheaply and accurately.
|
Flexibility and Speed. When you
need to build a system today that handles general queries over a few dozen
documents without training.
|
Compare classic FiD with long-context RAG across architecture, computational complexity, implementation burden, and lost-in-the-middle risk. section 5.1


