A decoder-only transformer is an autoregressive sequence model: it reads tokens left to right and predicts each next token from preceding context. This makes it suited to text generation, where future tokens do not yet exist. section 3.2.2
Distinguish decoder-only transformers from encoder-only transformers by their available context:
- Decoder-only models predict the next token using only preceding tokens. section 3.2.2
- Encoder-only models read a complete input bidirectionally, using future context to build richer representations for comparison and understanding. section 3.2.2
Context changes by architecture
When processing “attorney” in “The attorney filed the brief,” an encoder can use “filed” and “brief.” A decoder processes “attorney” without knowing what follows, because it is predicting the continuation.












