Masked attention

As explained in Machine Learning Algorithms in Depth

Masked attention restricts attention connections so that a position cannot use information from disallowed positions, such as future tokens.

Masked attention restricts attention connections so a decoder position cannot use disallowed information, especially future output tokens. It preserves causality during one-token-at-a-time generation. section 11.3

The masking mechanism operates as follows:

• Construct a mask matrix M containing zeros for permitted connections and negative infinity for prohibited connections. section 11.3
• Add M to the attention scores before applying softmax. section 11.3
• Let softmax preserve permitted connections while converting masked negative-infinity scores to zero, removing those connections from the output. section 11.3

Autoregressive translation

While generating a translated sentence, the decoder masks tokens that have not yet been generated, preventing later output words from influencing earlier predictions. section 11.3

Figure 11.10 Transformer architecture

Use the transformer architecture figure to locate masked multihead self-attention inside the decoder. section 11.3

How Machine Learning Algorithms in Depth covers it

How deep it goes

Defines the mask values and explains their softmax effect, but does not derive attention scores or implement the layer from scratch.

Examples

Uses autoregressive machine translation, where output tokens are generated one at a time.

What it assumes

Assumes familiarity with multihead self-attention, decoder layers, query-key-value attention, and softmax.

What you can do after

Understand how negative-infinity masking prevents future-token information from affecting decoder predictions.

How it connects

Connects masked attention with decoders, multihead self-attention, softmax, and encoder-decoder attention.

liveBookmanning.com

Read Machine Learning Algorithms in Depth on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Learn Generative AI with PyTorch
• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)Multi-head self-attention that restricts each position from accessing subsequent positions in the sequence.
• 

  liveBookmanning.com
  Sutskever's ListA sequence-modeling mask that prevents a position from attending to later positions.
• 

  liveBookmanning.com
  Transformers in ActionCausal masking restricts a sequence position from attending to later positions so predictions depend only on preceding context.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionCausal attention is attention constrained so that each position can use information from past and present positions but not future positions.

Compare how 6 books explain masked attention →

Related concepts

• transformer architectureused by · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
• attentionkind of · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
• decodepart of · Decoding is the process of transforming an encoded or internal representation into an interpretable or target-form output.
• decoder-only transformerused by · A decoder-only transformer is a Transformer architecture without a separate encoder that processes preceding context to predict or generate subsequent sequence elements.
• keyuses · In attention, a key is a vector representation of an input element that is compared with a query to determine relevance and attention weights for its associated value.
• queryuses · An attention query is a vector representation of the information sought by an input position, compared with key representations to determine which information to attend to.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning Algorithms in Depth. The section links point to the parts of the book it draws on.