Attention lets a neural network weigh which elements matter when representing or predicting an element. In a sentence, a word can attend to every word, including itself; higher relationship scores produce greater influence. This helps transformers connect parts of a sequence even when they are far apart. section 2.1.1 section 2.1
To calculate scaled dot-product attention:
- Represent inputs as query, key, and value vectors. Compare each query with the keys using dot products, then divide the scores by the square root of the key dimension. section 2.3.1
- If a mask is present, suppress masked scores before applying softmax. Softmax turns the remaining scores into attention weights. section 2.3.1
- Multiply the weights by the values to produce the attended representation. Split query, key, and value vectors across multiple heads so different heads can capture different aspects of the input. section 2.3.1
Listing 2.4 Calculating attention based on query, key, and value
def attention(query, key, value, mask=None, dropout=None):
d_k = query.size(-1)
scores = torch.matmul(query,
key.transpose(-2, -1)) / math.sqrt(d_k) #1
if mask is not None:
scores = scores.masked_fill(mask == 0, -1e9) #2
p_attn = nn.functional.softmax(scores, dim=-1) #3
if dropout is not None:
p_attn = dropout(p_attn)
return torch.matmul(p_attn, value), p_attn #4
The PyTorch listing makes the sequence concrete: score, optionally mask, normalize, then combine values. It also returns the weights, so they can be inspected separately from the attended result. section 2.3.1
From a German sentence to an English translation
Tokenize “Wie geht es dir” into [Wie, geht, es, dir], turn the tokens into word embeddings, and add positional encodings to retain their order. An encoder uses the resulting vectors to represent the German sentence; a decoder uses that representation to generate “How are you.” Attention supplies relationships between tokens, while positional encoding supplies order that attention alone does not account for. section 2.1.1 section 2.3 section 2.5
The same mechanism extends beyond words. A vision transformer treats image patches as tokens and uses self-attention to capture image features for classification. In a denoising U-Net, scaled dot-product attention instead weighs spatial regions in noisy images. Placed along the contracting and expansive paths, it helps the network emphasize useful features rather than redundant or irrelevant ones while predicting noise. section 3.0 section 7.1.1
Attention is one component, not a complete translator or image generator. It works alongside embeddings and positional encodings in transformers, and alongside convolutional layers and skip connections in a denoising U-Net. Its practical value is selective focus: the model can draw on relevant tokens or image regions when forming an output. section 2.1.1 section 7.1.1
