liveBook · Manning

Attention weight

An attention weight is a coefficient indicating how strongly an associated input or value contributes to an attention-based output or representation.

How 7 Manning books explain it

Timeless Algorithms, The Foundational Ideas

An attention weight is a normalized value indicating how strongly one element contributes to an attention-based representation.

How it is used
Uses attention weights to construct context-sensitive representations from weighted relationships among tokens.
How deep it goes
Works through scaled dot products, softmax normalization, and a weighted sum, while distinguishing the simplified calculation from a complete transformer.
Examples
Calculates weights for “trophy” and “box” in a pronoun-resolution example using simplified vectors.
What you can do after
Evaluate and understand how weights shape a contextual representation without treating them as a complete causal explanation of an output.

Read how Timeless Algorithms, The Foundational Ideas explains attention weight →

Build a DeepSeek Model (From Scratch)

An attention weight is a context-dependent coefficient that determines how strongly an input representation contributes to an attention output.

How it is used
Uses attention weights to construct context vectors for next-token prediction and motivate key-value caching.
How deep it goes
Traces scores through scaling, causal masking, softmax, and value multiplication, including matrix shapes and independent attention heads.
Examples
Traces a four-token attention matrix and a five-token newest-row calculation using “The next day is bright.”
What you can do after
Evaluate and understand how attention weights produce context vectors and why only the newest row is needed during generation.

Read how Build a DeepSeek Model (From Scratch) explains attention weight →

Build a Large Language Model (From Scratch)

A normalized relevance coefficient derived from attention scores that determines how much each value contributes to the context vector.

How it is used
Uses normalized relevance values to combine input or value vectors into context representations within self-attention.
How deep it goes
Derives weights from dot products through softmax and matrix operations, then extends them to projected queries and keys.
Examples
Demonstrates one query over six token vectors, row-wise normalization, vectorized weights, and a resulting three-dimensional context vector.
What you can do after
Build it from scratch with PyTorch using dot products, softmax, projected queries, keys, values, and matrix multiplication.

Read how Build a Large Language Model (From Scratch) explains attention weight →

Build a Text-to-Image Generator (from Scratch)

A normalized attention score that determines how strongly a value contributes to an attention output.

How it is used
Uses normalized query-key similarities to combine value vectors into contextually enriched token representations.
How deep it goes
Traces projections, scaled dot products, softmax, and value aggregation, then extends the process to multi-head attention.
Examples
Uses the contextual meaning of “bank,” PyTorch attention functions, masks, dropout, and four-head vision-transformer attention.
What you can do after
Build attention weights from query, key, and value tensors using scaled dot products, softmax, masking, and optional dropout.

Read how Build a Text-to-Image Generator (from Scratch) explains attention weight →

Build AI Drug Discovery Pipelines

An attention weight is a normalized numerical coefficient indicating how strongly an attention operation incorporates an associated value.

How it is used
Guides each decoder step toward the encoder information most relevant to the current protein-structure prediction.
How deep it goes
Explains weights through query-key comparisons and weighted sums, but does not detail their normalization or calculation formula.
Examples
Uses decoder positions, encoder hidden states, and local or distant amino-acid interactions as examples.
What you can do after
Understand how attention weights assemble position-specific context vectors during cross-attention.

Read how Build AI Drug Discovery Pipelines explains attention weight →

Machines that Think

An attention weight is a normalized value indicating how much one token attends to another token.

How it is used
Quantifies how strongly one token contributes context to another within a self-attention head.
How deep it goes
Explains row-wise softmax normalization and weighted value combination as parts of a single attention head.
Examples
Uses “rises” attending more to “sun” than “the” to illustrate weighted value contribution.
What you can do after
Understand how normalized weights guide value-vector contributions to contextual token representations.

Read how Machines that Think explains attention weight →

CUDA for LLMs

An attention weight is a normalized coefficient determining how much an attention value contributes to an output.

How it is used
Combines value vectors according to query-key similarity scores to produce attention outputs.
How deep it goes
Defines weights as softmaxed query-key scores and traces their N×N storage and HBM transfers, without detailing softmax internals.
Examples
Quantifies an 8,192-token matrix at 268 MB in float32 and cites over 65 GB per head at 128K context.
What you can do after
Evaluate attention’s intermediate-memory cost by estimating N×N matrices and their HBM traffic.

Read how CUDA for LLMs explains attention weight →

Also covered in

What to know first

Related concepts

Questions about attention weight

What is attention weight?

An attention weight is a coefficient indicating how strongly an associated input or value contributes to an attention-based output or representation.

Which Manning books explain attention weight?

Timeless Algorithms, The Foundational Ideas, Build a DeepSeek Model (From Scratch), Build a Large Language Model (From Scratch), Build a Text-to-Image Generator (from Scratch), Build AI Drug Discovery Pipelines, Machines that Think and CUDA for LLMs.

What should I know before learning attention weight?

Manning books that teach attention weight usually introduce softmax, transformer architecture, embedding and inner product first.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.