liveBook · Manning

Masked attention

Also called causal attention, causal masking

Masked attention is an attention mechanism that applies a mask to restrict which sequence positions can contribute to each position, often excluding future positions.

How 10 Manning books explain it

Learn Generative AI with PyTorch

How it is used
Uses causal attention in decoder-only Transformers to predict subsequent text and musical event tokens.
How deep it goes
Works through score masking, softmax, tensor shapes, and a PyTorch causal-attention implementation.
Examples
Uses a four-token prompt, a GPT-2XL implementation, and a music Transformer with eight attention heads.
What you can do after
Builds causal self-attention from scratch in PyTorch for a decoder-only Transformer.

Read how Learn Generative AI with PyTorch explains masked attention →

Build an Advanced RAG Application (From Scratch)

Masked multi-head self-attention is multi-head self-attention constrained so each position cannot attend to future sequence positions.

How it is used
Enforces causal next-token prediction within Transformer decoder layers for autoregressive text generation.
How deep it goes
Defines future-position blocking and explains its decoder role, without specifying mask construction or attention-score implementation.
Examples
Uses machine translation, text generation, prompts, and one-token-at-a-time next-word prediction as generation examples.
What you can do after
Evaluate/understand why decoder predictions exclude future tokens while retaining dependencies on prior generated tokens.

Read how Build an Advanced RAG Application (From Scratch) explains masked attention →

Build a Text-to-Image Generator (from Scratch)

Multi-head self-attention that restricts each position from accessing subsequent positions in the sequence.

How it is used
Restricts decoder self-attention so translation and caption generation predict tokens from previously available sequence positions.
How deep it goes
Explains the diagonal attention restriction, shifted targets, and decoder placement, but does not specify mask-construction code in the supplied excerpts.
Examples
Traces shifted caption tokens and their permitted attention positions, alongside an image-captioning decoder.
What you can do after
Evaluate/understand which input tokens each decoder position may use when predicting a caption’s next token.

Read how Build a Text-to-Image Generator (from Scratch) explains masked attention →

Sutskever's List

A sequence-modeling mask that prevents a position from attending to later positions.

How it is used
Enforces left-to-right information flow for autoregressive next-token prediction.
How deep it goes
Defines the mask’s information constraint and training role, but does not detail its implementation.
Examples
Uses next-token prediction at every sequence position as its training illustration.
What you can do after
Understand how causal masking prevents future information from influencing next-token predictions.

Read how Sutskever's List explains masked attention →

Transformers in Action

Causal masking restricts a sequence position from attending to later positions so predictions depend only on preceding context.

How it is used
Supports causal token-by-token generation in transformer decoders and decoder-only models.
How deep it goes
Defines the causal restriction and illustrates mask construction and application in TensorFlow, although the supplied application expression does not suppress future positions.
Examples
Illustrates a causal position mask with a TensorFlow masked multihead self-attention listing.
What you can do after
Evaluate and understand how causal masking restricts context during token generation.

Read how Transformers in Action explains masked attention →

Timeless Algorithms, The Foundational Ideas

Causal masking is a restriction on attention that prevents a sequence position from accessing subsequent positions.

How it is used
Enforces the rule that next-token predictions use only information available at each sequence position.
How deep it goes
Defines the mask with compatibility logits and softmax, then relates it to parallel training and sequential generation.
Examples
Uses the four-token prefix “The slower train has” to illustrate which earlier positions can attend to each token.
What you can do after
Understand how blocking future-token access preserves next-token prediction while allowing parallel computation across positions during training.

Read how Timeless Algorithms, The Foundational Ideas explains masked attention →

Building Specialized AI Systems

Causal masking prevents a sequence position from attending to tokens that occur later in the sequence.

How it is used
Supports decoder-only generation by restricting each position to earlier tokens.
How deep it goes
Defines causal masking and places it within decoder self-attention, but does not detail mask construction or tensor operations.
Examples
Contrasts decoder-only causal attention with encoder full-context attention and encoder–decoder sequence generation.
What you can do after
Evaluate whether decoder-only generation and its causal language-modeling objective fit a task.

Read how Building Specialized AI Systems explains masked attention →

Deep Learning with R, Third Edition

Causal attention is attention constrained so that each position can use information from past and present positions but not future positions.

How it is used
Prevents Transformer decoders from seeing future target tokens while predicting the next word or generating text.
How deep it goes
Explains information flow, displays the triangular mask, names the Keras option, and connects masking to key-value caching.
Examples
Uses a lower-triangular mask, Transformer translation decoding, and mini-GPT text generation as concrete examples.
What you can do after
Apply it via Keras MultiHeadAttention and reason about cached autoregressive generation.

Read how Deep Learning with R, Third Edition explains masked attention →

Build a DeepSeek Model (From Scratch)

A causal mask is an attention constraint that prevents each token from accessing later tokens in the sequence.

How it is used
Constrains self-attention during sequence processing so tokens cannot access future context.
How deep it goes
Descends from causal motivation through score masking, softmax behavior, and value aggregation, but stops short of code.
Examples
Uses a four-token attention matrix and the token “is” to illustrate preceding-context aggregation.
What you can do after
Evaluate causal masking choices and understand why −∞, rather than zero, blocks future-token probabilities.

Read how Build a DeepSeek Model (From Scratch) explains masked attention →

Machine Learning Algorithms in Depth

Masked attention restricts attention connections so that a position cannot use information from disallowed positions, such as future tokens.

How it is used
Preserves causality in decoder-based sequence generation.
How deep it goes
Defines the mask values and explains their softmax effect, but does not derive attention scores or implement the layer from scratch.
Examples
Uses autoregressive machine translation, where output tokens are generated one at a time.
What you can do after
Understand how negative-infinity masking prevents future-token information from affecting decoder predictions.

Read how Machine Learning Algorithms in Depth explains masked attention →

Also covered in

What to know first

Related concepts

Questions about masked attention

What is masked attention?

Masked attention is an attention mechanism that applies a mask to restrict which sequence positions can contribute to each position, often excluding future positions.

Which Manning books explain masked attention?

Learn Generative AI with PyTorch, Build an Advanced RAG Application (From Scratch), Build a Text-to-Image Generator (from Scratch), Sutskever's List, Transformers in Action, Timeless Algorithms, The Foundational Ideas, Building Specialized AI Systems, Deep Learning with R, Third Edition and 2 more.

What should I know before learning masked attention?

Manning books that teach masked attention usually introduce transformer architecture, attention, embedding and language model first.

What is masked attention a kind of?

Masked attention is a kind of attention. Attention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.

What is masked attention part of?

Masked attention is part of decoding strategy. A decoding strategy is a procedure for converting an encoded or internal representation into a target output, including selecting tokens during sequence generation.

What is masked attention also called?

Masked attention is also called causal attention or causal masking.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.