liveBook · Manning

Scaled-dot product attention

Also called SDPA

Scaled dot-product attention computes query–key dot products, scales them by the square root of the key dimension, applies softmax to obtain weights, and uses those weights to combine value vectors.

How 6 Manning books explain it

Build a Text-to-Image Generator (from Scratch)

Scaled dot-product attention computes attention weights from scaled dot products between query and key vectors.

How it is used
Computes relationships among sequence elements for transformers, vision transformers, and attention-equipped denoising U-Nets.
How deep it goes
Works through learned projections, scaled query–key scores, softmax weights, and value aggregation, then implements attention heads in PyTorch.
Examples
Uses German-to-English token embeddings, image-patch tokens, and flattened image pixels to illustrate attention across different sequences.
What you can do after
Build it from scratch using PyTorch attention heads and apply it within a transformer, vision transformer, or denoising U-Net.

Read how Build a Text-to-Image Generator (from Scratch) explains scaled-dot product attention →

Learn Generative AI with PyTorch

An attention method that computes similarity between queries and keys with dot products, scales them, and uses the result to weight values.

How it is used
Uses attention to relate sequence elements in a translation Transformer and image positions in a denoising U-Net.
How deep it goes
Explains projections, scaling, masking, softmax, and weighted values, with line-by-line implementations for text and images.
Examples
Works through “How are you?” with eight attention heads and a 64-by-64 image with four heads.
What you can do after
Builds attention from scratch for Transformer and denoising U-Net models.

Read how Learn Generative AI with PyTorch explains scaled-dot product attention →

Build AI Drug Discovery Pipelines

Scaled dot-product attention computes attention weights from query-key dot products divided by the square root of the key dimension and applies them to value vectors.

How it is used
Uses attention to capture long-range relationships between protein residues during representation learning.
How deep it goes
Explains scoring, scaling, softmax normalization, value aggregation, and final linear projection, but stops before implementation code.
Examples
Demonstrates three two-dimensional residue vectors and a distant cysteine pair with explicit scores, weights, and weighted values.
What you can do after
Evaluate/understand how attention weights stabilize training and combine context from sequence-distant residues.

Read how Build AI Drug Discovery Pipelines explains scaled-dot product attention →

Transformers in Action

Scaled dot-product attention computes query-key dot products, scales them by the square root of the key dimension, applies softmax, and uses the results to weight values.

How it is used
Introduces attention as a way to form sequence representations from relationships between positions in a transformer.
How deep it goes
Works through query, key, and value projections, score scaling, softmax, and weighted outputs with a simplified implementation.
Examples
Uses a five-position sequence to illustrate one output and four embeddings in a matrix implementation.
What you can do after
Build a simplified attention calculation from scratch using array and matrix operations.

Read how Transformers in Action explains scaled-dot product attention →

CUDA for LLMs

Scaled dot-product attention forms query-key scores, scales and normalizes them, and uses the resulting weights to combine values.

How it is used
Uses SDPA inside each head of multihead self-attention in a transformer.
How deep it goes
Explains the scaling and masking rationale and walks through a PyTorch head's forward computation.
Examples
Shows a PyTorch attention head with linear projections and a lower-triangular mask.
What you can do after
Apply it using PyTorch tensor operations to compute causally masked attention.

Read how CUDA for LLMs explains scaled-dot product attention →

Natural Language Processing in Action, Second Edition

Scaled dot-product attention divides query-key dot products by the square root of the key dimension before normalization.

How it is used
Stabilizes self-attention computations in transformer models, especially for high-dimensional embeddings and longer sequences.
How deep it goes
Explains projections, scaling, softmax normalization, weighted values, and matrix interpretation without expanding into implementation code.
Examples
Uses a toy encoder attention matrix to illustrate connections between words.
What you can do after
Build it from scratch by forming scaled query-key scores, normalizing them, and applying the resulting weights to values.

Read how Natural Language Processing in Action, Second Edition explains scaled-dot product attention →

Also covered in

What to know first

Related concepts

Questions about scaled-dot product attention

What is scaled-dot product attention?

Scaled dot-product attention computes query–key dot products, scales them by the square root of the key dimension, applies softmax to obtain weights, and uses those weights to combine value vectors.

Which Manning books explain scaled-dot product attention?

Build a Text-to-Image Generator (from Scratch), Learn Generative AI with PyTorch, Build AI Drug Discovery Pipelines, Transformers in Action, CUDA for LLMs and Natural Language Processing in Action, Second Edition.

What should I know before learning scaled-dot product attention?

Manning books that teach scaled-dot product attention usually introduce transformer architecture, inner product, softmax and attention first.

What is scaled-dot product attention a kind of?

Scaled-dot product attention is a kind of attention. Attention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.

What is scaled-dot product attention part of?

Scaled-dot product attention is part of attention head. An attention head is an independently parameterized attention computation that uses queries, keys, and values to weight and combine information across input elements.

What is scaled-dot product attention also called?

Scaled-dot product attention is also called SDPA.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.