liveBook · Manning

Masked attention

As explained in Build an Advanced RAG Application (From Scratch)

Masked multi-head self-attention is multi-head self-attention constrained so each position cannot attend to future sequence positions.

Masked multi-head self-attention is the decoder-side attention mechanism that preserves causal order during sequence generation. At any position, it blocks attention to later positions, so a next-token prediction uses only tokens already available in the sequence. section 2.2 section 2.3.2

The mask prevents a decoder from peeking at future tokens while training autoregressive tasks. This keeps training aligned with generation, where text is produced one token at a time and later tokens do not yet exist. section 2.2 section 5.1

Within a decoder layer, masked self-attention serves a distinct role alongside other components:

  • Masked self-attention captures dependencies among previously generated tokens, helping maintain coherent and contextually relevant output. section 2.3.2
  • Encoder-decoder attention can separately focus on relevant encoded input representations while output is being generated. section 2.3.2
  • Feed-forward networks process the attended information after the attention components. section 2.2

Next-token generation

For a prompt represented as a sequence of tokens, an autoregressive decoder predicts the next token from the preceding sequence. Masked self-attention enforces that each prediction is based on earlier generated tokens rather than future positions, supporting causal generation. section 2.3.2 section 5.1

Figure 5.1 Autoregressive Language Models predict the next word by processing a sequence of previous words. They use embedding, positional encoding, and multiple decoder blocks to generate a probability distribution for the next word.

This decoder-model view places masked self-attention among token embeddings, positional encodings, repeated decoder layers, and feed-forward networks used for next-word prediction. section 5.1

Masked attention is therefore not merely a context-selection step: it is the constraint that makes decoder generation causal. Combined with positional encodings, decoder layers, and feed-forward networks, it supports sequential processing and a probability distribution for the next token. section 5.1

How Build an Advanced RAG Application (From Scratch) covers it

How deep it goes
Defines future-position blocking and explains its decoder role, without specifying mask construction or attention-score implementation.
Examples
Uses machine translation, text generation, prompts, and one-token-at-a-time next-word prediction as generation examples.
What it assumes
Introduces masking through decoder generation while assuming familiarity with tokens, attention, and Transformer decoder components.
What you can do after
Evaluate/understand why decoder predictions exclude future tokens while retaining dependencies on prior generated tokens.
How it connects
Connects masked self-attention to Transformer decoders, encoder-decoder attention, feed-forward networks, positional encodings, and autoregressive language models.

Also explained in

Compare how 10 books explain masked attention →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build an Advanced RAG Application (From Scratch). The section links point to the parts of the book it draws on.