Scaled-dot product attention
Scaled dot-product attention computes query–key dot products, scales them by the square root of the key dimension, applies softmax to obtain weights, and uses those weights to combine value vectors.
Build a Text-to-Image Generator (from Scratch)
Scaled dot-product attention computes attention weights from scaled dot products between query and key vectors.
- How it is used
- Computes relationships among sequence elements for transformers, vision transformers, and attention-equipped denoising U-Nets.
- How deep it goes
- Works through learned projections, scaled query–key scores, softmax weights, and value aggregation, then implements attention heads in PyTorch.
- Examples
- Uses German-to-English token embeddings, image-patch tokens, and flattened image pixels to illustrate attention across different sequences.
- What you can do after
- Build it from scratch using PyTorch attention heads and apply it within a transformer, vision transformer, or denoising U-Net.
Read how Build a Text-to-Image Generator (from Scratch) explains scaled-dot product attention →
Learn Generative AI with PyTorch
An attention method that computes similarity between queries and keys with dot products, scales them, and uses the result to weight values.
- How it is used
- Uses attention to relate sequence elements in a translation Transformer and image positions in a denoising U-Net.
- How deep it goes
- Explains projections, scaling, masking, softmax, and weighted values, with line-by-line implementations for text and images.
- Examples
- Works through “How are you?” with eight attention heads and a 64-by-64 image with four heads.
- What you can do after
- Builds attention from scratch for Transformer and denoising U-Net models.
Read how Learn Generative AI with PyTorch explains scaled-dot product attention →
Build AI Drug Discovery Pipelines
Scaled dot-product attention computes attention weights from query-key dot products divided by the square root of the key dimension and applies them to value vectors.
- How it is used
- Uses attention to capture long-range relationships between protein residues during representation learning.
- How deep it goes
- Explains scoring, scaling, softmax normalization, value aggregation, and final linear projection, but stops before implementation code.
- Examples
- Demonstrates three two-dimensional residue vectors and a distant cysteine pair with explicit scores, weights, and weighted values.
- What you can do after
- Evaluate/understand how attention weights stabilize training and combine context from sequence-distant residues.
Read how Build AI Drug Discovery Pipelines explains scaled-dot product attention →
Transformers in Action
Scaled dot-product attention computes query-key dot products, scales them by the square root of the key dimension, applies softmax, and uses the results to weight values.
- How it is used
- Introduces attention as a way to form sequence representations from relationships between positions in a transformer.
- How deep it goes
- Works through query, key, and value projections, score scaling, softmax, and weighted outputs with a simplified implementation.
- Examples
- Uses a five-position sequence to illustrate one output and four embeddings in a matrix implementation.
- What you can do after
- Build a simplified attention calculation from scratch using array and matrix operations.
Read how Transformers in Action explains scaled-dot product attention →
CUDA for LLMs
Scaled dot-product attention forms query-key scores, scales and normalizes them, and uses the resulting weights to combine values.
- How it is used
- Uses SDPA inside each head of multihead self-attention in a transformer.
- How deep it goes
- Explains the scaling and masking rationale and walks through a PyTorch head's forward computation.
- Examples
- Shows a PyTorch attention head with linear projections and a lower-triangular mask.
- What you can do after
- Apply it using PyTorch tensor operations to compute causally masked attention.
Read how CUDA for LLMs explains scaled-dot product attention →
Natural Language Processing in Action, Second Edition
Scaled dot-product attention divides query-key dot products by the square root of the key dimension before normalization.
- How it is used
- Stabilizes self-attention computations in transformer models, especially for high-dimensional embeddings and longer sequences.
- How deep it goes
- Explains projections, scaling, softmax normalization, weighted values, and matrix interpretation without expanding into implementation code.
- Examples
- Uses a toy encoder attention matrix to illustrate connections between words.
- What you can do after
- Build it from scratch by forming scaled query-key scores, normalizing them, and applying the resulting weights to values.
Also covered in
- LLMs in Production
- Grokking AI Algorithms, Second Edition
- Deep Learning with PyTorch, Second Edition
What to know first
- transformer architectureA transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
- inner productAn inner product is a scalar-valued operation on two vectors that generalizes the dot product, which for real coordinate vectors is the sum of products of corresponding components.
- softmaxSoftmax is a function that exponentiates each value in a score vector and normalizes the results to produce positive values summing to one, forming a probability distribution.
- attentionAttention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
Related concepts
- attention headpart of · An attention head is an independently parameterized attention computation that uses queries, keys, and values to weight and combine information across input elements.
- Vision Transformerused by · A Vision Transformer is a Transformer-based neural-network architecture that processes images as sequences of tokenized regions using attention.
- attentionkind of · Attention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- queryuses · In an attention mechanism, a query is a vector representation of a sequence position that is compared with keys to determine relevance and weight associated values.
- inner productuses · An inner product is a scalar-valued operation on two vectors that generalizes the dot product, which for real coordinate vectors is the sum of products of corresponding components.
- query, key, and valueuses · Query, key, and value are attention representations that respectively express a matching request, matching features, and content to aggregate.
Questions about scaled-dot product attention
What is scaled-dot product attention?
Scaled dot-product attention computes query–key dot products, scales them by the square root of the key dimension, applies softmax to obtain weights, and uses those weights to combine value vectors.
Which Manning books explain scaled-dot product attention?
Build a Text-to-Image Generator (from Scratch), Learn Generative AI with PyTorch, Build AI Drug Discovery Pipelines, Transformers in Action, CUDA for LLMs and Natural Language Processing in Action, Second Edition.
What should I know before learning scaled-dot product attention?
Manning books that teach scaled-dot product attention usually introduce transformer architecture, inner product, softmax and attention first.
What is scaled-dot product attention a kind of?
Scaled-dot product attention is a kind of attention. Attention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
What is scaled-dot product attention part of?
Scaled-dot product attention is part of attention head. An attention head is an independently parameterized attention computation that uses queries, keys, and values to weight and combine information across input elements.
What is scaled-dot product attention also called?
Scaled-dot product attention is also called SDPA.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.





