liveBook · Manning

Lost in the middle

As explained in Retrieval Augmented Generation, The Foundational Ideas

A language-model performance degradation caused by relevant information being placed amid distracting or competing context.

Lost in the middle describes a language-model failure in which relevant information buried inside a long context receives less attention than information near the beginning or end. Irrelevant retrieved material can crowd out or obscure the useful passage, degrading answer quality. section 9.1.1 section 5.1

The effect appears in two common architecture patterns:

  • Passive retrieval adds documents for every prompt, including prompts that need no retrieval; irrelevant context consumes the budget and buries relevant evidence. section 9.1.1
  • Concatenating many documents into one sequence makes the model process a confused, expensive context; Transformer self-attention costs grow quadratically with sequence length. section 5.1
  • Fusion-in-Decoder reduces the risk by encoding documents independently and delaying fusion until generation, preserving distinct representations before synthesis. section 5.1

A mashed-together document set

Imagine ten different chapters mashed into one sequence, like ten half-shredded books glued together. Evidence at the beginning or end may receive more weight, while a relevant passage in the middle is overlooked. With 1,000 documents, prompt stuffing also becomes prohibitively slow and expensive. section 5.1

Table 5.1 Classic FiD vs. big context
Feature
Classic FiD (Encoder-Decoder)
Long-Context RAG (Decoder-Only)
Architecture
T5 / ATLAS. Encodes each document independently, then fuses them only during generation.
GPT / Llama. Concatenates all documents into one giant sequence and processes them simultaneously.
Complexity
Linear. Encoding 100 documents costs roughly 10 times the compute of encoding 10.
Quadratic. Self-attention costs grow squarely with length. Encoding 100 documents is significantly more expensive than encoding 10.
Lost in the Middle
Low Risk. Because the encoder processes every document in isolation, it creates a distinct representation for every passage before fusion occurs.
High Risk. Large Decoder-only models often ignore information buried in the middle of a massive prompt context.
Implementation
Harder. Requires fine-tuning and hosting your own model (e.g., HuggingFace transformers).
Easier. Often just an API call or a prompt template change.
Best For
Scale and Efficiency. When you have a fixed domain (e.g., medical records) and need to synthesize answers from 50–100+ documents cheaply and accurately.
Flexibility and Speed. When you need to build a system today that handles general queries over a few dozen documents without training.

Compare classic FiD with long-context RAG across architecture, computational complexity, implementation burden, and lost-in-the-middle risk. section 5.1

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains positional neglect, retrieval noise, quadratic attention costs, and independent encoding, but does not detail training procedures.
Examples
Uses mashed-together chapters, irrelevant retrieved documents, and 1,000-document queries to illustrate positional and computational failures.
What it assumes
Assumes basic familiarity with retrieval-augmented generation and Transformer attention, while explaining their relevance to context failures.
What you can do after
Diagnose/debug with it by recognizing when retrieval noise or concatenated long contexts obscure evidence, then compare independent encoding with prompt stuffing.
How it connects
Connects retrieval-augmented generation, Transformer self-attention, long-context models, prompt stuffing, and Fusion-in-Decoder.

Also explained in

Compare how 3 books explain lost in the middle →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.