liveBook · Manning

Self-attention

Self-attention is an attention mechanism that relates positions within a sequence, allowing each position to incorporate information from other positions in that sequence.

How 5 Manning books explain it

Timeless Algorithms, The Foundational Ideas

Self-attention forms representations of sequence elements by relating each element to others in the same sequence.

How it is used
Explains how transformers construct context through relationships among tokens rather than recurrent state propagation.
How deep it goes
Works through projections, scaled dot products, softmax, and weighted values, while distinguishing simplified calculations from full transformer behavior.
Examples
Uses pronoun resolution, factual recall, and autoregressive generation to illustrate distinct contributions of attention.
What you can do after
Evaluate how attention weights produce contextual representations and why a single attention calculation cannot explain complete model behavior.

Read how Timeless Algorithms, The Foundational Ideas explains self-attention →

Retrieval Augmented Generation, The Foundational Ideas

Self-attention is an attention mechanism that computes relationships among tokens by allowing each token to incorporate information from other tokens in the same sequence.

How it is used
Uses self-attention to explain contextual sentence representations and the cost of processing retrieved documents in one long sequence.
How deep it goes
Explains contextual word relationships and quadratic sequence-length cost, but does not derive the attention calculation.
Examples
Illustrates context with two meanings of “bank” and retrieval tradeoffs with support tickets and multiple retrieved documents.
What you can do after
Evaluate/understand when contextual representations help and when long-input self-attention costs favor independent document encoding.

Read how Retrieval Augmented Generation, The Foundational Ideas explains self-attention →

Transfer Learning for Natural Language Processing

Self-attention is an attention mechanism that relates positions within a sequence to other positions in that sequence.

How it is used
Uses self-attention to build contextual token representations in transformers for language understanding and translation.
How deep it goes
Works through learned query, key, and value vectors, scaled dot products, softmax coefficients, and parallel computation.
Examples
Uses a sentence about cells and a cell phone to illustrate contextual dependencies and visualize attention coefficients.
What you can do after
Apply it via bertviz to inspect attention outputs from a pretrained BERT encoder.

Read how Transfer Learning for Natural Language Processing explains self-attention →

Machines that Think

Self-attention is a sequence-processing mechanism in which each token assigns relevance to and gathers information from other tokens in the same sequence.

How it is used
Uses self-attention as the central mechanism for contextual sequence processing in Transformer language models.
How deep it goes
Explains learned query, key, and value projections, relevance comparisons, contextual representations, and causal masking without code or formal equations.
Examples
Uses a photograph analogy, the sentence “The sun rises in the east,” and attention matrices to illustrate relevance and masking.
What you can do after
Evaluate and understand how attention combines context and why generation restricts access to future tokens.

Read how Machines that Think explains self-attention →

CUDA for LLMs

Self-attention is an attention mechanism that computes relationships among positions in the same input sequence.

How it is used
Uses self-attention to gather information across sequence positions within transformer blocks.
How deep it goes
Works through projections, scaled scores, causal masking, softmax, and value aggregation, then places attention in a transformer block.
Examples
Uses a 16-sequence batch with length 64 and embedding size 128, and estimates one million dot products for 1,000 tokens.
What you can do after
Build causal multihead self-attention in PyTorch with linear projections, masking, softmax, and value aggregation.

Read how CUDA for LLMs explains self-attention →

What to know first

Related concepts

Questions about self-attention

What is self-attention?

Self-attention is an attention mechanism that relates positions within a sequence, allowing each position to incorporate information from other positions in that sequence.

Which Manning books explain self-attention?

Timeless Algorithms, The Foundational Ideas, Retrieval Augmented Generation, The Foundational Ideas, Transfer Learning for Natural Language Processing, Machines that Think and CUDA for LLMs.

What should I know before learning self-attention?

Manning books that teach self-attention usually introduce transformer architecture, attention, recurrent neural network and attention head first.

What kinds of self-attention are there?

Kinds of self-attention include bidirectional self-attention.

What is self-attention a kind of?

Self-attention is a kind of attention. Attention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.