An attention weight is a normalized score showing how strongly one value contributes to an attention output. Softmax converts scaled query-key similarity scores into weights whose total across the sequence is 100%. section 3.3.3 section 2.1.1
Compute attention weights through this sequence:
- Project the input embeddings into query, key, and value vectors. section 3.3.3
- Calculate query-key dot products and divide them by the square root of the key dimension, stabilizing the scores. section 2.1.1
- Apply softmax across the sequence to turn raw scores into normalized attention weights. section 2.3.1
- Multiply the weights by the value vectors to produce the attention output; multiple heads can learn different relationships. section 2.3.1
Listing 2.4 Calculating attention based on query, key, and value
def attention(query, key, value, mask=None, dropout=None):
d_k = query.size(-1)
scores = torch.matmul(query,
key.transpose(-2, -1)) / math.sqrt(d_k) #1
if mask is not None:
scores = scores.masked_fill(mask == 0, -1e9) #2
p_attn = nn.functional.softmax(scores, dim=-1) #3
if dropout is not None:
p_attn = dropout(p_attn)
return torch.matmul(p_attn, value), p_attn #4
The implementation returns both the weighted attention output and the attention weights, while supporting masks and dropout. section 2.3.1
Context-dependent weighting
In “I went fishing by the river in the morning, staying near the bank,” attention can connect “bank” with context such as “fish,” assigning greater weight to relevant tokens and producing a contextualized representation. section 2.1.1
