liveBook · Manning

Long-context RAG

As explained in Retrieval Augmented Generation, The Foundational Ideas

Long-context RAG is a retrieval-augmented generation pattern that concatenates many retrieved documents into a large model input context.

Long-context RAG concatenates many retrieved documents into one large decoder-only model input. It is a practical prototyping default, but long prompts can be costly, confusing, and vulnerable to hallucination or “lost in the middle” failures. section 5.1

Its central trade-offs are:

  • Self-attention cost grows quadratically with input length, making large collections expensive to compute. section 5.1
  • Information near the beginning or end may receive more attention than evidence buried in the middle. section 5.1
  • Fusion-in-Decoder encodes documents independently before generation, offering a more scalable alternative for very large document sets. section 5.1
Table 5.1 Classic FiD vs. big context
Feature
Classic FiD (Encoder-Decoder)
Long-Context RAG (Decoder-Only)
Architecture
T5 / ATLAS. Encodes each document independently, then fuses them only during generation.
GPT / Llama. Concatenates all documents into one giant sequence and processes them simultaneously.
Complexity
Linear. Encoding 100 documents costs roughly 10 times the compute of encoding 10.
Quadratic. Self-attention costs grow squarely with length. Encoding 100 documents is significantly more expensive than encoding 10.
Lost in the Middle
Low Risk. Because the encoder processes every document in isolation, it creates a distinct representation for every passage before fusion occurs.
High Risk. Large Decoder-only models often ignore information buried in the middle of a massive prompt context.
Implementation
Harder. Requires fine-tuning and hosting your own model (e.g., HuggingFace transformers).
Easier. Often just an API call or a prompt template change.
Best For
Scale and Efficiency. When you have a fixed domain (e.g., medical records) and need to synthesize answers from 50–100+ documents cheaply and accurately.
Flexibility and Speed. When you need to build a system today that handles general queries over a few dozen documents without training.

Compare Long-context RAG with classic Fusion-in-Decoder across architecture, complexity, middle-context risk, and implementation. section 5.1

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains architecture, quadratic cost, and lost-in-the-middle behavior, then contrasts independent encoding with FiD.
Examples
Uses fifty-document prompt stuffing, one thousand-document processing, and the mashed-books analogy to illustrate scale and confusion.
What you can do after
Evaluate when prompt concatenation is acceptable and when scalable independent document encoding is preferable.
How it connects
Connects retrieval-augmented generation with decoder-only models, self-attention, hallucination, and Fusion-in-Decoder.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.