Dot-product attention
Dot-product attention computes query–key relevance scores using inner products and uses normalized scores to form weighted combinations of value representations.
Natural Language Processing in Action, Second Edition
Dot-product attention scores queries and keys with scalar products and uses those scores to weight values.
- How it is used
- Implements attention in transformers through optimized matrix multiplications over packed query, key, and value matrices.
- How deep it goes
- Explains projections, scaling, softmax normalization, and weighted values, while stopping before multi-head attention details.
- Examples
- Uses a toy attention matrix to represent connections between words in an input sequence.
- What you can do after
- Evaluate how scaled query-key scores become normalized weights and weighted value outputs.
Read how Natural Language Processing in Action, Second Edition explains dot-product attention →
Deep Learning with R, Third Edition
Dot-product attention computes attention relevance scores from the dot products of query-like and source-like vectors and uses them to form a weighted representation.
- How it is used
- Uses relevance-weighted source representations to help sequence models predict target tokens.
- How deep it goes
- Explains dot products, softmax normalization, weighted sums, sequence-shaped score matrices, and tensor implementation.
- Examples
- Demonstrates translation prediction, source-target score matrices, and image retrieval through query-key matching.
- What you can do after
- Build it from scratch using tensor contractions, softmax scores, and weighted source representations.
Read how Deep Learning with R, Third Edition explains dot-product attention →
Related concepts
- attentionkind of · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.