Masked attention
Masked attention is an attention mechanism in which a mask restricts which positions can influence each other, often preventing sequence positions from accessing future tokens.
Learn Generative AI with PyTorch
- How it is used
- Restricts attention to current and earlier tokens for next-token prediction in text and musical-event generation.
- How deep it goes
- Works through tensor shapes, score masking, softmax weights, and a PyTorch attention module rather than stopping at a definition.
- Examples
- Uses a four-token prompt, a lower-triangular attention matrix, GPT-2XL tensor shapes, and a musical-event Transformer.
- What you can do after
- Build it from scratch in PyTorch as causal self-attention and use it in a decoder-only Transformer.
Read how Learn Generative AI with PyTorch explains masked attention →
Build a Text-to-Image Generator (from Scratch)
Multi-head self-attention that restricts each position from accessing subsequent positions in the sequence.
- How it is used
- Restricts decoder self-attention so translation and caption generation predict tokens from previously available sequence positions.
- How deep it goes
- Explains the diagonal attention restriction, shifted targets, and decoder placement, but does not specify mask-construction code in the supplied excerpts.
- Examples
- Traces shifted caption tokens and their permitted attention positions, alongside an image-captioning decoder.
- What you can do after
- Evaluate/understand which input tokens each decoder position may use when predicting a caption’s next token.
Read how Build a Text-to-Image Generator (from Scratch) explains masked attention →
Sutskever's List
A sequence-modeling mask that prevents a position from attending to later positions.
- How it is used
- Enforces left-to-right information flow for autoregressive next-token prediction.
- How deep it goes
- Defines the mask’s information constraint and training role, but does not detail its implementation.
- Examples
- Uses next-token prediction at every sequence position as its training illustration.
- What you can do after
- Understand how causal masking prevents future information from influencing next-token predictions.
Transformers in Action
Causal masking restricts a sequence position from attending to later positions so predictions depend only on preceding context.
- How it is used
- Constrains decoder predictions to available context during autoregressive, token-by-token generation.
- How deep it goes
- Defines the causal restriction and shows mask construction and an attention-weight update, but stops short of a verified working implementation.
- Examples
- Uses an attention-masking figure and a TensorFlow listing that constructs a position mask for decoder-only self-attention.
- What you can do after
- Diagnose/debug with it by checking whether an attention-weight update actually suppresses access to future positions.
Deep Learning with R, Third Edition
Causal attention is attention constrained so that each position can use information from past and present positions but not future positions.
- How it is used
- Prevents Transformer decoders from seeing future target tokens while predicting the next word or generating text.
- How deep it goes
- Explains information flow, displays the triangular mask, names the Keras option, and connects masking to key-value caching.
- Examples
- Uses a lower-triangular mask, Transformer translation decoding, and mini-GPT text generation as concrete examples.
- What you can do after
- Apply it via Keras MultiHeadAttention and reason about cached autoregressive generation.
Read how Deep Learning with R, Third Edition explains masked attention →
Machine Learning Algorithms in Depth
Masked attention restricts attention connections so that a position cannot use information from disallowed positions, such as future tokens.
- How it is used
- Preserves causality in decoder-based sequence generation.
- How deep it goes
- Defines the mask values and explains their softmax effect, but does not derive attention scores or implement the layer from scratch.
- Examples
- Uses autoregressive machine translation, where output tokens are generated one at a time.
- What you can do after
- Understand how negative-infinity masking prevents future-token information from affecting decoder predictions.
Read how Machine Learning Algorithms in Depth explains masked attention →
Related concepts
- transformer architectureused by · A neural-network architecture that models relationships among sequence or structured-input elements primarily through attention and related transformations.
- attentionkind of · Attention is a neural-network mechanism that computes input-dependent weights over relationships among representations and uses them to combine information relevant to a computation.
- decoder-only transformerused by · A decoder-only transformer is a Transformer architecture without a separate encoder that processes preceding context to predict or generate subsequent sequence elements.
- decodepart of · Decoding is the process of interpreting an encoded or internal representation to recover or produce corresponding data, symbols, or output.
- keyuses · In attention, a key is a vector representation of an input element that is compared with a query to determine relevance and attention weights for its associated value.
- queryuses · In attention, a query is a vector representation of what a position seeks or evaluates, compared with keys to determine which information to retrieve or emphasize.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.