How deeply does Deep Learning with R, Third Edition cover masked attention?

Explains information flow, displays the triangular mask, names the Keras option, and connects masking to key-value caching.

Masked attention

As explained in Deep Learning with R, Third Edition

Causal attention is attention constrained so that each position can use information from past and present positions but not future positions.

Masked attention restricts self-attention so each position uses only permitted context. Causal attention allows a token to attend to itself and earlier tokens, preventing future-token leakage during next-token prediction. section 15.3.3

Lower-triangular causal mask

For five target positions, a causal mask is: 1 0 0 0 0; 1 1 0 0 0; 1 1 1 0 0; 1 1 1 1 0; 1 1 1 1 1. Row i specifies which positions token i may attend to, reproducing one-directional information flow without a bidirectional shortcut. section 15.3.3

Use causal masking when:

Training a Transformer decoder to predict the next word without exposing the label it must predict. section 15.3.3
Generating text autoregressively, where each new position may depend on past and present tokens but not future tokens. section 16.2.3
Calling Keras MultiHeadAttention with use_causal_mask to create the required lower-triangular restriction. section 15.3.3

The mask also enables efficient generation: because future positions cannot change past key and value vectors, implementations can cache those vectors and compute one position at a time. section 16.2.3

How Deep Learning with R, Third Edition covers it

Cover of Deep Learning with R, Third Edition

liveBook manning.com

Read Deep Learning with R, Third Edition on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Learn Generative AI with PyTorch
liveBook manning.com

Build a Text-to-Image Generator (from Scratch)Multi-head self-attention that restricts each position from accessing subsequent positions in the sequence.
liveBook manning.com

Sutskever's ListA sequence-modeling mask that prevents a position from attending to later positions.
liveBook manning.com

Transformers in ActionCausal masking restricts a sequence position from attending to later positions so predictions depend only on preceding context.
liveBook manning.com

Machine Learning Algorithms in DepthMasked attention restricts attention connections so that a position cannot use information from disallowed positions, such as future tokens.

Compare how 6 books explain masked attention →

Related concepts

transformer architectureused by · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
attentionkind of · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
decodepart of · Decoding is the process of transforming an encoded or internal representation into an interpretable or target-form output.
decoder-only transformerused by · A decoder-only transformer is a Transformer architecture without a separate encoder that processes preceding context to predict or generate subsequent sequence elements.
keyuses · In attention, a key is a vector representation of an input element that is compared with a query to determine relevance and attention weights for its associated value.
queryuses · An attention query is a vector representation of the information sought by an input position, compared with key representations to determine which information to attend to.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with R, Third Edition. The section links point to the parts of the book it draws on.