Scaled-dot product attention
Scaled dot-product attention computes query-key dot products, divides them by the square root of the key dimension, applies softmax, and uses the resulting weights to combine value vectors.
Build a Text-to-Image Generator (from Scratch)
Scaled dot-product attention computes attention weights from scaled dot products between query and key vectors.
- How it is used
- Computes relationships among sequence elements for transformers, vision transformers, and attention-equipped denoising U-Nets.
- How deep it goes
- Works through learned projections, scaled query–key scores, softmax weights, and value aggregation, then implements attention heads in PyTorch.
- Examples
- Uses German-to-English token embeddings, image-patch tokens, and flattened image pixels to illustrate attention across different sequences.
- What you can do after
- Build it from scratch using PyTorch attention heads and apply it within a transformer, vision transformer, or denoising U-Net.
Read how Build a Text-to-Image Generator (from Scratch) explains scaled-dot product attention →
Learn Generative AI with PyTorch
An attention method that computes similarity between queries and keys with dot products, scales them, and uses the result to weight values.
- How it is used
- Computes relationships among text tokens in Transformers and among pixel positions in denoising U-Nets.
- How deep it goes
- Works through query, key, and value projections, score scaling, softmax, masking, and value weighting before extending the calculation to multiple heads.
- Examples
- Uses a short text phrase with 256-dimensional embeddings and a 64 × 64 image-feature example flattened into pixel positions.
- What you can do after
- Build it from scratch in PyTorch as a Transformer attention function and adapt the calculation to flattened image features.
Read how Learn Generative AI with PyTorch explains scaled-dot product attention →
Transformers in Action
Scaled dot-product attention computes query-key dot products, scales them by the square root of the key dimension, applies softmax, and uses the results to weight values.
- How it is used
- Produces sequence representations by weighting values according to query–key comparisons within self-attention.
- How deep it goes
- Works through the calculation and a simplified implementation, including the reason for scaling.
- Examples
- Uses a five-input output calculation and a NumPy implementation with projected embeddings.
- What you can do after
- Build it from scratch using matrix projections, scaled dot products, softmax, and value weighting.
Read how Transformers in Action explains scaled-dot product attention →
Natural Language Processing in Action, Second Edition
Scaled dot-product attention divides query-key dot products by the square root of the key dimension before normalization.
- How it is used
- Stabilizes self-attention computations in transformer models, especially for high-dimensional embeddings and longer sequences.
- How deep it goes
- Explains projections, scaling, softmax normalization, weighted values, and matrix interpretation without expanding into implementation code.
- Examples
- Uses a toy encoder attention matrix to illustrate connections between words.
- What you can do after
- Build it from scratch by forming scaled query-key scores, normalizing them, and applying the resulting weights to values.
Related concepts
- Vision Transformerused by · A Vision Transformer is a Transformer-based neural-network architecture that processes images as sequences of tokenized regions using attention.
- attentionkind of · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
- queryuses · An attention query is a vector representation of the information sought by an input position, compared with key representations to determine which information to attend to.
- inner productuses · An inner product is a scalar-valued operation on two vectors that combines their corresponding components, summing their products and, for complex vectors, using conjugation.
- query, key, and valueuses · Query, key, and value are attention representations that respectively express a matching request, matching features, and content to aggregate.
- U-Netused by · U-Net is an encoder-decoder neural-network architecture with downsampling and upsampling paths connected by skip connections for combining features at multiple spatial scales.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.