Masked attention

As explained in Build a Text-to-Image Generator (from Scratch)

Multi-head self-attention that restricts each position from accessing subsequent positions in the sequence.

Masked attention is self-attention that prevents a sequence position from accessing later positions. Standard self-attention lets each token attend to every token; a causal attention mask instead leaves each token able to attend only to itself and earlier tokens. In a transformer decoder, this lets the model predict the next token without seeing future input tokens. section 4.1.1 section 2.1.2

To use the mask for next-token training:

• Prepare an input sequence and a target shifted one position ahead. For a caption, [<start>, two, bears, fight] is paired with [two, bears, fight, <end>]. section 4.1.1
• Apply the causal mask to self-attention weights, hiding positions above the main diagonal. At each input position, the decoder can use its current token and earlier tokens, but not later ones. section 4.1.1
• Compare the decoder’s predictions with the shifted targets using cross-entropy loss. The targets supply the training signal; they are not fed to the decoder as its input sequence. section 4.1.2

Figure 4.3 How the causal attention mask works

The causal-mask diagram contrasts attention to every sequence position with attention restricted to the current position and those before it. section 4.1.1

In an encoder–decoder transformer, masked multi-head self-attention is the first sublayer of each decoder block. Cross-attention then connects the decoder state to the encoder output, followed by a feed-forward sublayer. For image captioning, the encoder supplies image embeddings while masked self-attention keeps caption generation dependent on preceding caption tokens. Word embeddings and positional encodings form the decoder input. section 4.1.2 section 4.3.2

What each caption position may use

With [<start>, two, bears, fight] as input, the first position uses <start> to predict two. The next uses <start> and two to predict bears, without accessing the later input tokens. The final position can use the full input sequence to predict <end>. The same restriction applies within the decoder layers during training. section 4.1.1 section 4.1.2

How Build a Text-to-Image Generator (from Scratch) covers it

How deep it goes

Explains the diagonal attention restriction, shifted targets, and decoder placement, but does not specify mask-construction code in the supplied excerpts.

Examples

Traces shifted caption tokens and their permitted attention positions, alongside an image-captioning decoder.

What it assumes

Builds on self-attention and token sequences while explaining the causal restriction and shifted next-token targets.

What you can do after

Evaluate/understand which input tokens each decoder position may use when predicting a caption’s next token.

How it connects

Connects causal attention masks to multi-head self-attention, cross-attention, positional encoding, and cross-entropy loss.

liveBookmanning.com

Read Build a Text-to-Image Generator (from Scratch) on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Learn Generative AI with PyTorch
• 

  liveBookmanning.com
  Sutskever's ListA sequence-modeling mask that prevents a position from attending to later positions.
• 

  liveBookmanning.com
  Transformers in ActionCausal masking restricts a sequence position from attending to later positions so predictions depend only on preceding context.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionCausal attention is attention constrained so that each position can use information from past and present positions but not future positions.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthMasked attention restricts attention connections so that a position cannot use information from disallowed positions, such as future tokens.

Compare how 6 books explain masked attention →

Related concepts

• transformer architectureused by · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
• attentionkind of · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
• decodepart of · Decoding is the process of transforming an encoded or internal representation into an interpretable or target-form output.
• decoder-only transformerused by · A decoder-only transformer is a Transformer architecture without a separate encoder that processes preceding context to predict or generate subsequent sequence elements.
• keyuses · In attention, a key is a vector representation of an input element that is compared with a query to determine relevance and attention weights for its associated value.
• queryuses · An attention query is a vector representation of the information sought by an input position, compared with key representations to determine which information to attend to.

This explainer was generated by Manning's knowledge graph from the text of Build a Text-to-Image Generator (from Scratch). The section links point to the parts of the book it draws on.