How deeply does Sutskever's List cover masked attention?

Defines the mask’s information constraint and training role, but does not detail its implementation.

Masked attention

A sequence-modeling mask that prevents a position from attending to later positions.

In autoregressive text generation, each token is predicted from what came before. A causal mask enforces this direction: a token can attend only to preceding tokens, preventing information from the future from leaking into its prediction. section 5.1

During training, the model predicts the next token at every position in a sequence simultaneously. The mask preserves that left-to-right prediction task while allowing the sequence to provide many training examples. section 5.1

How Sutskever's List covers it

liveBook manning.com

Read Sutskever's List on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Learn Generative AI with PyTorch
liveBook manning.com

Build a Text-to-Image Generator (from Scratch)Multi-head self-attention that restricts each position from accessing subsequent positions in the sequence.
liveBook manning.com

Transformers in ActionCausal masking restricts a sequence position from attending to later positions so predictions depend only on preceding context.
liveBook manning.com

Deep Learning with R, Third EditionCausal attention is attention constrained so that each position can use information from past and present positions but not future positions.
liveBook manning.com

Machine Learning Algorithms in DepthMasked attention restricts attention connections so that a position cannot use information from disallowed positions, such as future tokens.

Compare how 6 books explain masked attention →

Related concepts

transformer architectureused by · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
attentionkind of · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
decodepart of · Decoding is the process of transforming an encoded or internal representation into an interpretable or target-form output.
decoder-only transformerused by · A decoder-only transformer is a Transformer architecture without a separate encoder that processes preceding context to predict or generate subsequent sequence elements.
keyuses · In attention, a key is a vector representation of an input element that is compared with a query to determine relevance and attention weights for its associated value.
queryuses · An attention query is a vector representation of the information sought by an input position, compared with key representations to determine which information to attend to.

This explainer was generated by Manning's knowledge graph from the text of Sutskever's List. The section links point to the parts of the book it draws on.