Self-attention
Self-attention is an attention mechanism that relates positions within a sequence, allowing each position to incorporate information from other positions in that sequence.
Timeless Algorithms, The Foundational Ideas
Self-attention forms representations of sequence elements by relating each element to others in the same sequence.
- How it is used
- Explains how transformers construct context through relationships among tokens rather than recurrent state propagation.
- How deep it goes
- Works through projections, scaled dot products, softmax, and weighted values, while distinguishing simplified calculations from full transformer behavior.
- Examples
- Uses pronoun resolution, factual recall, and autoregressive generation to illustrate distinct contributions of attention.
- What you can do after
- Evaluate how attention weights produce contextual representations and why a single attention calculation cannot explain complete model behavior.
Read how Timeless Algorithms, The Foundational Ideas explains self-attention →
Retrieval Augmented Generation, The Foundational Ideas
Self-attention is an attention mechanism that computes relationships among tokens by allowing each token to incorporate information from other tokens in the same sequence.
- How it is used
- Uses self-attention to explain contextual sentence representations and the cost of processing retrieved documents in one long sequence.
- How deep it goes
- Explains contextual word relationships and quadratic sequence-length cost, but does not derive the attention calculation.
- Examples
- Illustrates context with two meanings of “bank” and retrieval tradeoffs with support tickets and multiple retrieved documents.
- What you can do after
- Evaluate/understand when contextual representations help and when long-input self-attention costs favor independent document encoding.
Read how Retrieval Augmented Generation, The Foundational Ideas explains self-attention →
Transfer Learning for Natural Language Processing
Self-attention is an attention mechanism that relates positions within a sequence to other positions in that sequence.
- How it is used
- Uses self-attention to build contextual token representations in transformers for language understanding and translation.
- How deep it goes
- Works through learned query, key, and value vectors, scaled dot products, softmax coefficients, and parallel computation.
- Examples
- Uses a sentence about cells and a cell phone to illustrate contextual dependencies and visualize attention coefficients.
- What you can do after
- Apply it via bertviz to inspect attention outputs from a pretrained BERT encoder.
Read how Transfer Learning for Natural Language Processing explains self-attention →
Machines that Think
Self-attention is a sequence-processing mechanism in which each token assigns relevance to and gathers information from other tokens in the same sequence.
- How it is used
- Uses self-attention as the central mechanism for contextual sequence processing in Transformer language models.
- How deep it goes
- Explains learned query, key, and value projections, relevance comparisons, contextual representations, and causal masking without code or formal equations.
- Examples
- Uses a photograph analogy, the sentence “The sun rises in the east,” and attention matrices to illustrate relevance and masking.
- What you can do after
- Evaluate and understand how attention combines context and why generation restricts access to future tokens.
CUDA for LLMs
Self-attention is an attention mechanism that computes relationships among positions in the same input sequence.
- How it is used
- Uses self-attention to gather information across sequence positions within transformer blocks.
- How deep it goes
- Works through projections, scaled scores, causal masking, softmax, and value aggregation, then places attention in a transformer block.
- Examples
- Uses a 16-sequence batch with length 64 and embedding size 128, and estimates one million dot products for 1,000 tokens.
- What you can do after
- Build causal multihead self-attention in PyTorch with linear projections, masking, softmax, and value aggregation.
What to know first
- transformer architectureA transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
- attentionAttention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- recurrent neural networkA recurrent neural network is a neural network that processes ordered inputs while carrying and updating an internal state across successive steps.
- attention headAn attention head is an independently parameterized attention computation that uses queries, keys, and values to weight and combine information across input elements.
Related concepts
- attentionkind of · Attention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- queryuses · In an attention mechanism, a query is a vector representation of a sequence position that is compared with keys to determine relevance and weight associated values.
- transformer architectureused by · A transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
- attention weightuses · An attention weight is a coefficient indicating how strongly an associated input or value contributes to an attention-based output or representation.
- inner productuses · An inner product is a scalar-valued operation on two vectors that generalizes the dot product, which for real coordinate vectors is the sum of products of corresponding components.
Questions about self-attention
What is self-attention?
Self-attention is an attention mechanism that relates positions within a sequence, allowing each position to incorporate information from other positions in that sequence.
Which Manning books explain self-attention?
Timeless Algorithms, The Foundational Ideas, Retrieval Augmented Generation, The Foundational Ideas, Transfer Learning for Natural Language Processing, Machines that Think and CUDA for LLMs.
What should I know before learning self-attention?
Manning books that teach self-attention usually introduce transformer architecture, attention, recurrent neural network and attention head first.
What kinds of self-attention are there?
Kinds of self-attention include bidirectional self-attention.
What is self-attention a kind of?
Self-attention is a kind of attention. Attention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.




