Attention weight
An attention weight is a coefficient indicating how strongly an associated input or value contributes to an attention-based output or representation.
Timeless Algorithms, The Foundational Ideas
An attention weight is a normalized value indicating how strongly one element contributes to an attention-based representation.
- How it is used
- Uses attention weights to construct context-sensitive representations from weighted relationships among tokens.
- How deep it goes
- Works through scaled dot products, softmax normalization, and a weighted sum, while distinguishing the simplified calculation from a complete transformer.
- Examples
- Calculates weights for “trophy” and “box” in a pronoun-resolution example using simplified vectors.
- What you can do after
- Evaluate and understand how weights shape a contextual representation without treating them as a complete causal explanation of an output.
Read how Timeless Algorithms, The Foundational Ideas explains attention weight →
Build a DeepSeek Model (From Scratch)
An attention weight is a context-dependent coefficient that determines how strongly an input representation contributes to an attention output.
- How it is used
- Uses attention weights to construct context vectors for next-token prediction and motivate key-value caching.
- How deep it goes
- Traces scores through scaling, causal masking, softmax, and value multiplication, including matrix shapes and independent attention heads.
- Examples
- Traces a four-token attention matrix and a five-token newest-row calculation using “The next day is bright.”
- What you can do after
- Evaluate and understand how attention weights produce context vectors and why only the newest row is needed during generation.
Read how Build a DeepSeek Model (From Scratch) explains attention weight →
Build a Large Language Model (From Scratch)
A normalized relevance coefficient derived from attention scores that determines how much each value contributes to the context vector.
- How it is used
- Uses normalized relevance values to combine input or value vectors into context representations within self-attention.
- How deep it goes
- Derives weights from dot products through softmax and matrix operations, then extends them to projected queries and keys.
- Examples
- Demonstrates one query over six token vectors, row-wise normalization, vectorized weights, and a resulting three-dimensional context vector.
- What you can do after
- Build it from scratch with PyTorch using dot products, softmax, projected queries, keys, values, and matrix multiplication.
Read how Build a Large Language Model (From Scratch) explains attention weight →
Build a Text-to-Image Generator (from Scratch)
A normalized attention score that determines how strongly a value contributes to an attention output.
- How it is used
- Uses normalized query-key similarities to combine value vectors into contextually enriched token representations.
- How deep it goes
- Traces projections, scaled dot products, softmax, and value aggregation, then extends the process to multi-head attention.
- Examples
- Uses the contextual meaning of “bank,” PyTorch attention functions, masks, dropout, and four-head vision-transformer attention.
- What you can do after
- Build attention weights from query, key, and value tensors using scaled dot products, softmax, masking, and optional dropout.
Read how Build a Text-to-Image Generator (from Scratch) explains attention weight →
Build AI Drug Discovery Pipelines
An attention weight is a normalized numerical coefficient indicating how strongly an attention operation incorporates an associated value.
- How it is used
- Guides each decoder step toward the encoder information most relevant to the current protein-structure prediction.
- How deep it goes
- Explains weights through query-key comparisons and weighted sums, but does not detail their normalization or calculation formula.
- Examples
- Uses decoder positions, encoder hidden states, and local or distant amino-acid interactions as examples.
- What you can do after
- Understand how attention weights assemble position-specific context vectors during cross-attention.
Read how Build AI Drug Discovery Pipelines explains attention weight →
Machines that Think
An attention weight is a normalized value indicating how much one token attends to another token.
- How it is used
- Quantifies how strongly one token contributes context to another within a self-attention head.
- How deep it goes
- Explains row-wise softmax normalization and weighted value combination as parts of a single attention head.
- Examples
- Uses “rises” attending more to “sun” than “the” to illustrate weighted value contribution.
- What you can do after
- Understand how normalized weights guide value-vector contributions to contextual token representations.
CUDA for LLMs
An attention weight is a normalized coefficient determining how much an attention value contributes to an output.
- How it is used
- Combines value vectors according to query-key similarity scores to produce attention outputs.
- How deep it goes
- Defines weights as softmaxed query-key scores and traces their N×N storage and HBM transfers, without detailing softmax internals.
- Examples
- Quantifies an 8,192-token matrix at 268 MB in float32 and cites over 65 GB per head at 128K context.
- What you can do after
- Evaluate attention’s intermediate-memory cost by estimating N×N matrices and their HBM traffic.
Also covered in
- Grokking AI Algorithms, Second Edition
- Inside Deep Learning: Math, Algorithms, Models
- Crack Any Codebase with AIIn early access (MEAP)
- Deep Learning with PyTorch, Second Edition
- Knowledge Graphs and LLMs in Action
What to know first
- softmaxSoftmax is a function that exponentiates each value in a score vector and normalizes the results to produce positive values summing to one, forming a probability distribution.
- transformer architectureA transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
- embeddingAn embedding is a numerical vector representation of an item in a feature space, encoding selected properties or relationships for computational tasks such as comparison or retrieval.
- inner productAn inner product is a scalar-valued operation on two vectors that generalizes the dot product, which for real coordinate vectors is the sum of products of corresponding components.
Related concepts
- self-attentionused by · Self-attention is an attention mechanism that relates positions within a sequence, allowing each position to incorporate information from other positions in that sequence.
- softmaxproduced by · Softmax is a function that exponentiates each value in a score vector and normalizes the results to produce positive values summing to one, forming a probability distribution.
- attention scoreproduced by · An attention score is a numerical value expressing the relevance or compatibility between elements and used to determine their relative influence in an attention mechanism.
- valueuses · In attention, a value is a projected vector representation of content that is combined with other values according to attention weights to produce an output.
- context matrixproduces · A context matrix contains attention outputs for all sequence positions, with each row formed by weighting value vectors.
- context vectorproduces · A context vector is a numerical vector that summarizes information from relevant input context for a model’s prediction or subsequent processing.
Questions about attention weight
What is attention weight?
An attention weight is a coefficient indicating how strongly an associated input or value contributes to an attention-based output or representation.
Which Manning books explain attention weight?
Timeless Algorithms, The Foundational Ideas, Build a DeepSeek Model (From Scratch), Build a Large Language Model (From Scratch), Build a Text-to-Image Generator (from Scratch), Build AI Drug Discovery Pipelines, Machines that Think and CUDA for LLMs.
What should I know before learning attention weight?
Manning books that teach attention weight usually introduce softmax, transformer architecture, embedding and inner product first.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.






