liveBook · Manning

Decoder-only transformer

As explained in Retrieval Augmented Generation, The Foundational Ideas

A decoder-only model generates text from a single autoregressive sequence without a separate encoder for input representation.

A decoder-only transformer processes one continuous prompt sequence and generates its continuation, without separately encoding input before writing. GPT, LLaMA, and Mistral use this pattern for modern text generation. section 5.1

For retrieval-augmented generation, the architecture creates practical tradeoffs:

  • Long-context RAG concatenates documents into one sequence, so self-attention processes them together and computation grows quadratically with sequence length. section 5.1
  • Compared with encoder-decoder FiD, decoder-only processing risks ignoring information buried in the middle of a massive prompt. section 5.1
  • Encoder-decoder models remain a practical alternative when documents should be encoded independently before generation. section 5.1
Table 5.1 Classic FiD vs. big context
Feature
Classic FiD (Encoder-Decoder)
Long-Context RAG (Decoder-Only)
Architecture
T5 / ATLAS. Encodes each document independently, then fuses them only during generation.
GPT / Llama. Concatenates all documents into one giant sequence and processes them simultaneously.
Complexity
Linear. Encoding 100 documents costs roughly 10 times the compute of encoding 10.
Quadratic. Self-attention costs grow squarely with length. Encoding 100 documents is significantly more expensive than encoding 10.
Lost in the Middle
Low Risk. Because the encoder processes every document in isolation, it creates a distinct representation for every passage before fusion occurs.
High Risk. Large Decoder-only models often ignore information buried in the middle of a massive prompt context.
Implementation
Harder. Requires fine-tuning and hosting your own model (e.g., HuggingFace transformers).
Easier. Often just an API call or a prompt template change.
Best For
Scale and Efficiency. When you have a fixed domain (e.g., medical records) and need to synthesize answers from 50–100+ documents cheaply and accurately.
Flexibility and Speed. When you need to build a system today that handles general queries over a few dozen documents without training.

Compare decoder-only long-context RAG with classic encoder-decoder FiD across architecture, complexity, lost-in-the-middle risk, and implementation. section 5.1

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Contrasts sequence processing, attention complexity, and middle-context failure risk, but does not descend into decoder internals.
Examples
Uses GPT, LLaMA, Mistral, and concatenated multi-document prompts as concrete decoder-only examples.
What you can do after
Evaluate decoder-only and encoder-decoder architectures for retrieval systems using context risk, complexity, and implementation tradeoffs.
How it connects
Connects decoder-only transformers with self-attention, long-context RAG, encoder-decoder architectures, and the lost-in-the-middle problem.

Also explained in

Compare how 13 books explain decoder-only transformer →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.