Masked attention
Masked attention is an attention mechanism that applies a mask to restrict which sequence positions can contribute to each position, often excluding future positions.
Learn Generative AI with PyTorch
- How it is used
- Uses causal attention in decoder-only Transformers to predict subsequent text and musical event tokens.
- How deep it goes
- Works through score masking, softmax, tensor shapes, and a PyTorch causal-attention implementation.
- Examples
- Uses a four-token prompt, a GPT-2XL implementation, and a music Transformer with eight attention heads.
- What you can do after
- Builds causal self-attention from scratch in PyTorch for a decoder-only Transformer.
Read how Learn Generative AI with PyTorch explains masked attention →
Build an Advanced RAG Application (From Scratch)
Masked multi-head self-attention is multi-head self-attention constrained so each position cannot attend to future sequence positions.
- How it is used
- Enforces causal next-token prediction within Transformer decoder layers for autoregressive text generation.
- How deep it goes
- Defines future-position blocking and explains its decoder role, without specifying mask construction or attention-score implementation.
- Examples
- Uses machine translation, text generation, prompts, and one-token-at-a-time next-word prediction as generation examples.
- What you can do after
- Evaluate/understand why decoder predictions exclude future tokens while retaining dependencies on prior generated tokens.
Read how Build an Advanced RAG Application (From Scratch) explains masked attention →
Build a Text-to-Image Generator (from Scratch)
Multi-head self-attention that restricts each position from accessing subsequent positions in the sequence.
- How it is used
- Restricts decoder self-attention so translation and caption generation predict tokens from previously available sequence positions.
- How deep it goes
- Explains the diagonal attention restriction, shifted targets, and decoder placement, but does not specify mask-construction code in the supplied excerpts.
- Examples
- Traces shifted caption tokens and their permitted attention positions, alongside an image-captioning decoder.
- What you can do after
- Evaluate/understand which input tokens each decoder position may use when predicting a caption’s next token.
Read how Build a Text-to-Image Generator (from Scratch) explains masked attention →
Sutskever's List
A sequence-modeling mask that prevents a position from attending to later positions.
- How it is used
- Enforces left-to-right information flow for autoregressive next-token prediction.
- How deep it goes
- Defines the mask’s information constraint and training role, but does not detail its implementation.
- Examples
- Uses next-token prediction at every sequence position as its training illustration.
- What you can do after
- Understand how causal masking prevents future information from influencing next-token predictions.
Transformers in Action
Causal masking restricts a sequence position from attending to later positions so predictions depend only on preceding context.
- How it is used
- Supports causal token-by-token generation in transformer decoders and decoder-only models.
- How deep it goes
- Defines the causal restriction and illustrates mask construction and application in TensorFlow, although the supplied application expression does not suppress future positions.
- Examples
- Illustrates a causal position mask with a TensorFlow masked multihead self-attention listing.
- What you can do after
- Evaluate and understand how causal masking restricts context during token generation.
Timeless Algorithms, The Foundational Ideas
Causal masking is a restriction on attention that prevents a sequence position from accessing subsequent positions.
- How it is used
- Enforces the rule that next-token predictions use only information available at each sequence position.
- How deep it goes
- Defines the mask with compatibility logits and softmax, then relates it to parallel training and sequential generation.
- Examples
- Uses the four-token prefix “The slower train has” to illustrate which earlier positions can attend to each token.
- What you can do after
- Understand how blocking future-token access preserves next-token prediction while allowing parallel computation across positions during training.
Read how Timeless Algorithms, The Foundational Ideas explains masked attention →
Building Specialized AI Systems
Causal masking prevents a sequence position from attending to tokens that occur later in the sequence.
- How it is used
- Supports decoder-only generation by restricting each position to earlier tokens.
- How deep it goes
- Defines causal masking and places it within decoder self-attention, but does not detail mask construction or tensor operations.
- Examples
- Contrasts decoder-only causal attention with encoder full-context attention and encoder–decoder sequence generation.
- What you can do after
- Evaluate whether decoder-only generation and its causal language-modeling objective fit a task.
Read how Building Specialized AI Systems explains masked attention →
Deep Learning with R, Third Edition
Causal attention is attention constrained so that each position can use information from past and present positions but not future positions.
- How it is used
- Prevents Transformer decoders from seeing future target tokens while predicting the next word or generating text.
- How deep it goes
- Explains information flow, displays the triangular mask, names the Keras option, and connects masking to key-value caching.
- Examples
- Uses a lower-triangular mask, Transformer translation decoding, and mini-GPT text generation as concrete examples.
- What you can do after
- Apply it via Keras MultiHeadAttention and reason about cached autoregressive generation.
Read how Deep Learning with R, Third Edition explains masked attention →
Build a DeepSeek Model (From Scratch)
A causal mask is an attention constraint that prevents each token from accessing later tokens in the sequence.
- How it is used
- Constrains self-attention during sequence processing so tokens cannot access future context.
- How deep it goes
- Descends from causal motivation through score masking, softmax behavior, and value aggregation, but stops short of code.
- Examples
- Uses a four-token attention matrix and the token “is” to illustrate preceding-context aggregation.
- What you can do after
- Evaluate causal masking choices and understand why −∞, rather than zero, blocks future-token probabilities.
Read how Build a DeepSeek Model (From Scratch) explains masked attention →
Machine Learning Algorithms in Depth
Masked attention restricts attention connections so that a position cannot use information from disallowed positions, such as future tokens.
- How it is used
- Preserves causality in decoder-based sequence generation.
- How deep it goes
- Defines the mask values and explains their softmax effect, but does not derive attention scores or implement the layer from scratch.
- Examples
- Uses autoregressive machine translation, where output tokens are generated one at a time.
- What you can do after
- Understand how negative-infinity masking prevents future-token information from affecting decoder predictions.
Read how Machine Learning Algorithms in Depth explains masked attention →
Also covered in
- Build a Large Language Model (From Scratch)
- Natural Language Processing in Action, Second Edition
- Hugging Face in Action
- LLMs in Production
- Deep Learning with PyTorch, Second Edition
- Deep Learning with Python, Third Edition
- Machines that ThinkIn early access (MEAP)
What to know first
- transformer architectureA transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
- attentionAttention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- embeddingAn embedding is a numerical vector representation of an item in a feature space, encoding selected properties or relationships for computational tasks such as comparison or retrieval.
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
Related concepts
- transformer architectureused by · A transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
- attentionkind of · Attention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- decoder-only transformerused by · A decoder-only transformer is a Transformer language model with decoder layers and no separate encoder, generating sequence elements autoregressively from preceding context.
- decoding strategypart of · A decoding strategy is a procedure for converting an encoded or internal representation into a target output, including selecting tokens during sequence generation.
- keyuses · In an attention mechanism, a key is a vector representation of an input element that is compared with a query to determine the relevance of its associated value.
- queryuses · In an attention mechanism, a query is a vector representation of a sequence position that is compared with keys to determine relevance and weight associated values.
Questions about masked attention
What is masked attention?
Masked attention is an attention mechanism that applies a mask to restrict which sequence positions can contribute to each position, often excluding future positions.
Which Manning books explain masked attention?
Learn Generative AI with PyTorch, Build an Advanced RAG Application (From Scratch), Build a Text-to-Image Generator (from Scratch), Sutskever's List, Transformers in Action, Timeless Algorithms, The Foundational Ideas, Building Specialized AI Systems, Deep Learning with R, Third Edition and 2 more.
What should I know before learning masked attention?
Manning books that teach masked attention usually introduce transformer architecture, attention, embedding and language model first.
What is masked attention a kind of?
Masked attention is a kind of attention. Attention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
What is masked attention part of?
Masked attention is part of decoding strategy. A decoding strategy is a procedure for converting an encoded or internal representation into a target output, including selecting tokens during sequence generation.
What is masked attention also called?
Masked attention is also called causal attention or causal masking.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.









