Attention score
An attention score is a numerical value expressing the relevance or compatibility between elements and used to determine their relative influence in an attention mechanism.
Build a DeepSeek Model (From Scratch)
An attention score is a scalar measuring the relevance between a query and a key before normalization into attention weights.
- How it is used
- Uses scores to determine token relevance and motivate efficient autoregressive attention.
- How deep it goes
- Works through matrix shapes, masking, normalization, and separate content and positional score paths.
- Examples
- Traces a four-token sequence through a 4×4 score matrix and derives the newest token’s score row.
- What you can do after
- Evaluate/understand how query–key products become attention weights and how a new token’s score row is calculated.
Read how Build a DeepSeek Model (From Scratch) explains attention score →
Graph Neural Networks in Action
An unnormalized learned score quantifying the importance of one connected node's features to another node.
- How it is used
- Weights neighbor contributions during GAT message passing for tasks such as node classification and link prediction.
- How deep it goes
- Defines learned pairwise scores, softmax normalization, and feature aggregation without presenting a rigorous mathematical derivation.
- Examples
- Uses connected-node feature vectors, fraud-detection edges, GAT equations, and PyTorch Geometric GATConv implementations.
- What you can do after
- Apply it via PyTorch Geometric to understand or implement attention-weighted graph aggregation.
Read how Graph Neural Networks in Action explains attention score →
Build AI Drug Discovery Pipelines
An attention score is a numerical compatibility value between a query and a key before normalization into an attention weight.
- How it is used
- Explains how cross-attention selects relevant encoder information during protein-sequence decoding.
- How deep it goes
- Defines query-key compatibility and follows normalization into weights, stopping short of specifying a scoring formula.
- Examples
- Uses protein positions 45 and 312 to illustrate local and long-range encoder relevance.
- What you can do after
- Evaluate/understand how raw compatibility values become normalized contributions to a decoder context vector.
Read how Build AI Drug Discovery Pipelines explains attention score →
Deep Learning with R, Third Edition
An attention score is a numerical value measuring the relevance of one vector or sequence position to another in an attention calculation.
- How it is used
- Uses relevance scores to selectively retrieve source-sequence information while predicting target tokens in sequence-to-sequence models.
- How deep it goes
- Explains dot products, softmax normalization, weighted sums, batched score matrices, and their role in attention computation.
- Examples
- Uses translation vectors, a normalized weighted-sum pseudocode example, and an R tensor implementation with sequence score matrices.
- What you can do after
- Build dot-product attention that normalizes source relevance and combines source vectors into context representations.
Read how Deep Learning with R, Third Edition explains attention score →
Build a Text-to-Image Generator (from Scratch)
An attention score is a numerical measure of the relevance between a query and a key.
- How it is used
- Measures query-key relevance inside scaled dot-product attention before normalization.
- How deep it goes
- Derives scores from dot products, scaling, softmax, and value aggregation, but defers detailed multi-head class internals.
- Examples
- Uses book recommendation vectors, “Wie geht es dir,” and vision-transformer attention heads as concrete examples.
- What you can do after
- Apply attention scores by implementing an AttentionHead in PyTorch with scaled dot products and softmax.
Read how Build a Text-to-Image Generator (from Scratch) explains attention score →
The AI Pocket Book
A numerical value used by an attention mechanism to weight the relevance of one element relative to others.
- How it is used
- Disambiguates tokens by weighting the relevance of surrounding context during contextualization.
- How deep it goes
- Explains token weighting, projections, learned matrices, and embedding updates but defers the detailed mathematical operations.
- Examples
- Uses “dog’s bark” to show how relative token relevance resolves meaning.
- What you can do after
- Evaluate how token relevance guides contextualized embeddings in a language model.
What to know first
- inner productAn inner product is a scalar-valued operation on two vectors that generalizes the dot product, which for real coordinate vectors is the sum of products of corresponding components.
- transformer architectureA transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
- embeddingAn embedding is a numerical vector representation of an item in a feature space, encoding selected properties or relationships for computational tasks such as comparison or retrieval.
- recurrent neural networkA recurrent neural network is a neural network that processes ordered inputs while carrying and updating an internal state across successive steps.
Related concepts
- inner productproduced by · An inner product is a scalar-valued operation on two vectors that generalizes the dot product, which for real coordinate vectors is the sum of products of corresponding components.
- attention weightproduces · An attention weight is a coefficient indicating how strongly an associated input or value contributes to an attention-based output or representation.
- queryproduced by · In an attention mechanism, a query is a vector representation of a sequence position that is compared with keys to determine relevance and weight associated values.
- softmaxused by · Softmax is a function that exponentiates each value in a score vector and normalizes the results to produce positive values summing to one, forming a probability distribution.
- attentionproduced by · Attention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- keyproduced by · In an attention mechanism, a key is a vector representation of an input element that is compared with a query to determine the relevance of its associated value.
Questions about attention score
What is attention score?
An attention score is a numerical value expressing the relevance or compatibility between elements and used to determine their relative influence in an attention mechanism.
Which Manning books explain attention score?
Build a DeepSeek Model (From Scratch), Graph Neural Networks in Action, Build AI Drug Discovery Pipelines, Deep Learning with R, Third Edition, Build a Text-to-Image Generator (from Scratch) and The AI Pocket Book.
What should I know before learning attention score?
Manning books that teach attention score usually introduce inner product, transformer architecture, embedding and recurrent neural network first.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.





