Attention weight
An attention weight is a computed coefficient, often normalized, that determines how strongly an input element or value contributes to an attention-based representation.
Build a Large Language Model (From Scratch)
A normalized relevance coefficient derived from attention scores that determines how much each value contributes to the context vector.
- How it is used
- Uses normalized relevance values to combine input or value vectors into context representations within self-attention.
- How deep it goes
- Derives weights from dot products through softmax and matrix operations, then extends them to projected queries and keys.
- Examples
- Demonstrates one query over six token vectors, row-wise normalization, vectorized weights, and a resulting three-dimensional context vector.
- What you can do after
- Build it from scratch with PyTorch using dot products, softmax, projected queries, keys, values, and matrix multiplication.
Read how Build a Large Language Model (From Scratch) explains attention weight →
Build a Text-to-Image Generator (from Scratch)
A normalized attention score that determines how strongly a value contributes to an attention output.
- How it is used
- Uses normalized query-key similarities to combine value vectors into contextually enriched token representations.
- How deep it goes
- Traces projections, scaled dot products, softmax, and value aggregation, then extends the process to multi-head attention.
- Examples
- Uses the contextual meaning of “bank,” PyTorch attention functions, masks, dropout, and four-head vision-transformer attention.
- What you can do after
- Build attention weights from query, key, and value tensors using scaled dot products, softmax, masking, and optional dropout.
Read how Build a Text-to-Image Generator (from Scratch) explains attention weight →
Related concepts
- softmaxproduced by · Softmax is a function that exponentiates a vector of scores and normalizes the results into nonnegative values that sum to one.
- attention scoreproduced by · An attention score is a numerical compatibility or relevance value computed between elements, such as a query and key, before normalization into attention weights.
- valueuses · In attention, a value is a vector representation of input content that is combined with other values according to attention weights to produce an output.
- context vectorproduces · A context vector is a numerical representation that summarizes information from an input or surrounding context relevant to a model’s prediction or subsequent processing.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.