Masked multi-head self-attention is the decoder-side attention mechanism that preserves causal order during sequence generation. At any position, it blocks attention to later positions, so a next-token prediction uses only tokens already available in the sequence. section 2.2 section 2.3.2
The mask prevents a decoder from peeking at future tokens while training autoregressive tasks. This keeps training aligned with generation, where text is produced one token at a time and later tokens do not yet exist. section 2.2 section 5.1
Within a decoder layer, masked self-attention serves a distinct role alongside other components:
- Masked self-attention captures dependencies among previously generated tokens, helping maintain coherent and contextually relevant output. section 2.3.2
- Encoder-decoder attention can separately focus on relevant encoded input representations while output is being generated. section 2.3.2
- Feed-forward networks process the attended information after the attention components. section 2.2
Next-token generation
For a prompt represented as a sequence of tokens, an autoregressive decoder predicts the next token from the preceding sequence. Masked self-attention enforces that each prediction is based on earlier generated tokens rather than future positions, supporting causal generation. section 2.3.2 section 5.1
Figure 5.1 Autoregressive Language Models predict the next word by processing a sequence of previous words. They use embedding, positional encoding, and multiple decoder blocks to generate a probability distribution for the next word.
This decoder-model view places masked self-attention among token embeddings, positional encodings, repeated decoder layers, and feed-forward networks used for next-word prediction. section 5.1
Masked attention is therefore not merely a context-selection step: it is the constraint that makes decoder generation causal. Combined with positional encodings, decoder layers, and feed-forward networks, it supports sequential processing and a probability distribution for the next token. section 5.1









