A character embedding stays the same wherever that character appears. But a character can play a different role depending on its neighbors: the “a” in “car” and “care” is a simple example. Attention lets a model adjust which parts of a sequence matter at each position, rather than relying only on static embeddings. section 9.5
Think of attention as a movable spotlight. At each position, three learned representations help decide where it points:
- A query represents what this position is looking for. section 9.5
- A key represents what another position offers for matching. section 9.5
- A value carries the information retrieved when a query and key match. section 9.5
In self-attention, the positions being compared come from the same sequence. Learned linear transformations produce queries, keys, and values from the input embeddings. Query–key dot products give relevance scores; softmax turns those scores into weights. Each position’s output is a weighted combination of value vectors, so it can draw more information from relevant positions. Scaled dot-product attention also scales the scores by the key dimension. section 9.5.1 section 9.12
A small PyTorch calculation
With query, key, and value tensors made by separate linear transformations, the core calculation is `weights = F.softmax(query @ key.T, dim=-1)` followed by `output = weights @ value`. This small example shows unscaled dot-product attention; PyTorch also provides `torch.nn.functional.scaled_dot_product_attention` for the scaled operation. section 9.5.1 section 9.12
Listing 9.2 attention.ipynb
# In[]:
torch.manual_seed(0)
x = torch.rand(2, 3)
query = F.linear(x, weight=torch.rand(3, 3),
bias=torch.rand(3)) #1
key = F.linear(x, weight=torch.rand(3, 3), bias=torch.rand(3))
value = F.linear(x, weight=torch.rand(3, 3), bias=torch.rand(3))
def dot_product_attention_single(q, k, v):
attn_weights = q @ k.T #2
attn_weights = F.softmax(attn_weights, dim=-1) #3
output = attn_weights @ v #4
return output
dot_product_attention_single(query, key, value)
# Out[]:
tensor([[1.3667, 0.6913, 1.0614],
[1.3718, 0.6930, 1.0627]])
The listing constructs two input positions, projects them into queries, keys, and values, then computes softmax weights and the resulting outputs. section 9.5.1
Attention helps transformers capture relationships across a sequence for contextually relevant text generation. The same idea applies to images in a Vision Transformer: image patches become tokens, and self-attention lets a classification token gather information from them. A further exercise is to modify a self-attention implementation into cross-attention. section 9.12 section 9.9 section 9.11
