liveBook · Manning

Masked attention

As explained in Timeless Algorithms, The Foundational Ideas

Causal masking is a restriction on attention that prevents a sequence position from accessing subsequent positions.

Causal masking prevents each sequence position from attending to later positions. In a four-token sequence, earlier positions can use only the tokens up to and including themselves. Adding the mask to attention compatibility logits assigns blocked positions a value of −∞; softmax then gives them zero weight. section 12.2.3

Smasked=S+M

The masked scores are the original compatibility scores plus the mask: allowed relationships remain available, while blocked future positions receive −∞. section 12.2.3

During generation, a predicted token is appended and becomes available to the next prediction. During training, causal masking preserves the next-token objective while allowing positions to be calculated in parallel; generation itself remains sequential. Masking is one part of attention, not a complete explanation of a transformer’s behavior. section 12.2.3 section 12.2 section 12.3.2

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines the mask with compatibility logits and softmax, then relates it to parallel training and sequential generation.
Examples
Uses the four-token prefix “The slower train has” to illustrate which earlier positions can attend to each token.
What it assumes
Assumes attention scores and softmax are available, while defining how a mask removes future positions from usable context.
What you can do after
Understand how blocking future-token access preserves next-token prediction while allowing parallel computation across positions during training.
How it connects
Connects attention and softmax to autoregressive generation, next-token prediction, and parallel transformer training.

Also explained in

Compare how 10 books explain masked attention →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.