Scaled-dot product attention

As explained in Natural Language Processing in Action, Second Edition

Scaled dot-product attention divides query-key dot products by the square root of the key dimension before normalization.

Scaled-dot product attention turns relationships between input embeddings into weighted context vectors. It projects embeddings into query, key, and value matrices, divides query-key products by √dₖ, normalizes them with softmax, and uses the scores to combine values. section 9.1.2

The scaling step matters because:

• High-dimensional query-key products can produce small softmax gradients; multiplying by 1/√dₖ improves numerical stability. section 9.1.2
• The softmax produces positive scores that sum to one, creating weights for the value vectors. section 9.1.2
• The resulting Q × K matrix represents connections between words and supports efficient matrix-multiplication implementations. section 9.1.2

Equation 9.1 Self-attention outputs

Use the self-attention equation to trace the scaling, normalization, and weighted-value steps together. section 9.1.2

How Natural Language Processing in Action, Second Edition covers it

How deep it goes

Explains projections, scaling, softmax normalization, weighted values, and matrix interpretation without expanding into implementation code.

Examples

Uses a toy encoder attention matrix to illustrate connections between words.

What it assumes

Assumes familiarity with embedding vectors, dot products, matrix multiplication, and softmax normalization.

What you can do after

Build it from scratch by forming scaled query-key scores, normalizing them, and applying the resulting weights to values.

How it connects

Connects query, key, value matrices with self-attention, softmax, embeddings, transformers, and matrix multiplication.

liveBookmanning.com

Read Natural Language Processing in Action, Second Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)Scaled dot-product attention computes attention weights from scaled dot products between query and key vectors.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchAn attention method that computes similarity between queries and keys with dot products, scales them, and uses the result to weight values.
• 

  liveBookmanning.com
  Transformers in ActionScaled dot-product attention computes query-key dot products, scales them by the square root of the key dimension, applies softmax, and uses the results to weight values.

Compare how 4 books explain scaled-dot product attention →

Related concepts

• Vision Transformerused by · A Vision Transformer is a Transformer-based neural-network architecture that processes images as sequences of tokenized regions using attention.
• attentionkind of · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
• queryuses · An attention query is a vector representation of the information sought by an input position, compared with key representations to determine which information to attend to.
• inner productuses · An inner product is a scalar-valued operation on two vectors that combines their corresponding components, summing their products and, for complex vectors, using conjugation.
• query, key, and valueuses · Query, key, and value are attention representations that respectively express a matching request, matching features, and content to aggregate.
• U-Netused by · U-Net is an encoder-decoder neural-network architecture with downsampling and upsampling paths connected by skip connections for combining features at multiple spatial scales.

This explainer was generated by Manning's knowledge graph from the text of Natural Language Processing in Action, Second Edition. The section links point to the parts of the book it draws on.