Attention score
An attention score is a numerical compatibility or relevance value computed between elements, such as a query and key, before normalization into attention weights.
Graph Neural Networks in Action
An unnormalized learned score quantifying the importance of one connected node's features to another node.
- How it is used
- Weights neighbor contributions during GAT message passing for tasks such as node classification and link prediction.
- How deep it goes
- Defines learned pairwise scores, softmax normalization, and feature aggregation without presenting a rigorous mathematical derivation.
- Examples
- Uses connected-node feature vectors, fraud-detection edges, GAT equations, and PyTorch Geometric GATConv implementations.
- What you can do after
- Apply it via PyTorch Geometric to understand or implement attention-weighted graph aggregation.
Read how Graph Neural Networks in Action explains attention score →
Deep Learning with R, Third Edition
An attention score is a numerical value measuring the relevance of one vector or sequence position to another in an attention calculation.
- How it is used
- Uses relevance scores to selectively retrieve source-sequence information while predicting target tokens in sequence-to-sequence models.
- How deep it goes
- Explains dot products, softmax normalization, weighted sums, batched score matrices, and their role in attention computation.
- Examples
- Uses translation vectors, a normalized weighted-sum pseudocode example, and an R tensor implementation with sequence score matrices.
- What you can do after
- Build dot-product attention that normalizes source relevance and combines source vectors into context representations.
Read how Deep Learning with R, Third Edition explains attention score →
Build a Text-to-Image Generator (from Scratch)
An attention score is a numerical measure of the relevance between a query and a key.
- How it is used
- Measures query-key relevance inside scaled dot-product attention before normalization.
- How deep it goes
- Derives scores from dot products, scaling, softmax, and value aggregation, but defers detailed multi-head class internals.
- Examples
- Uses book recommendation vectors, “Wie geht es dir,” and vision-transformer attention heads as concrete examples.
- What you can do after
- Apply attention scores by implementing an AttentionHead in PyTorch with scaled dot products and softmax.
Read how Build a Text-to-Image Generator (from Scratch) explains attention score →
The AI Pocket Book
A numerical value used by an attention mechanism to weight the relevance of one element relative to others.
- How it is used
- Disambiguates tokens by weighting the relevance of surrounding context during contextualization.
- How deep it goes
- Explains token weighting, projections, learned matrices, and embedding updates but defers the detailed mathematical operations.
- Examples
- Uses “dog’s bark” to show how relative token relevance resolves meaning.
- What you can do after
- Evaluate how token relevance guides contextualized embeddings in a language model.
Related concepts
- inner productproduced by · An inner product is a scalar-valued operation on two vectors that combines their corresponding components, summing their products and, for complex vectors, using conjugation.
- attention weightproduces · An attention weight is a computed coefficient, often normalized, that determines how strongly an input element or value contributes to an attention-based representation.
- queryproduced by · An attention query is a vector representation of the information sought by an input position, compared with key representations to determine which information to attend to.
- attentionproduced by · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
- keyproduced by · In attention, a key is a vector representation of an input element that is compared with a query to determine relevance and attention weights for its associated value.
- softmaxuses · Softmax is a function that exponentiates a vector of scores and normalizes the results into nonnegative values that sum to one.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.