A decoder-only transformer processes one continuous prompt sequence and generates its continuation, without separately encoding input before writing. GPT, LLaMA, and Mistral use this pattern for modern text generation. section 5.1
For retrieval-augmented generation, the architecture creates practical tradeoffs:
- Long-context RAG concatenates documents into one sequence, so self-attention processes them together and computation grows quadratically with sequence length. section 5.1
- Compared with encoder-decoder FiD, decoder-only processing risks ignoring information buried in the middle of a massive prompt. section 5.1
- Encoder-decoder models remain a practical alternative when documents should be encoded independently before generation. section 5.1
Table 5.1 Classic FiD vs. big context
|
Feature
|
Classic FiD (Encoder-Decoder)
|
Long-Context RAG (Decoder-Only)
|
|
Architecture
|
T5 / ATLAS. Encodes each document
independently, then fuses them only during generation.
|
GPT / Llama. Concatenates all
documents into one giant sequence and processes them simultaneously.
|
|
Complexity
|
Linear. Encoding 100 documents
costs roughly 10 times the compute of encoding 10.
|
Quadratic. Self-attention costs
grow squarely with length. Encoding 100 documents is significantly more
expensive than encoding 10.
|
|
Lost in the Middle
|
Low Risk. Because the encoder
processes every document in isolation, it creates a distinct representation
for every passage before fusion occurs.
|
High Risk. Large Decoder-only
models often ignore information buried in the middle of a massive prompt
context.
|
|
Implementation
|
Harder. Requires fine-tuning and
hosting your own model (e.g., HuggingFace transformers).
|
Easier. Often just an API call or
a prompt template change.
|
|
Best For
|
Scale and Efficiency. When you
have a fixed domain (e.g., medical records) and need to synthesize answers
from 50–100+ documents cheaply and accurately.
|
Flexibility and Speed. When you
need to build a system today that handles general queries over a few dozen
documents without training.
|
Compare decoder-only long-context RAG with classic encoder-decoder FiD across architecture, complexity, lost-in-the-middle risk, and implementation. section 5.1












