A value is the content attention retrieves after deciding which input positions matter. Unlike a static character embedding, it contributes context-dependent information to the output. section 9.5
In self-attention, a learnable linear transformation converts each input embedding into a value vector. For position n, attention combines all value vectors using weights that describe how much position n should focus on position m. section 9.5.1
The value participates in attention as follows:
- Transform every input embedding into a value vector with a learnable linear layer. section 9.5.1
- Compute attention weights from query-key relationships, then normalize them with softmax. section 9.5.1
- Produce each output by taking the weighted sum of the value vectors. section 9.5.1
Values in a small attention calculation
A PyTorch example applies F.linear separately to create query, key, and value tensors, then computes attention weights with q @ k.T, normalizes them with F.softmax, and returns attn_weights @ v. section 9.5.1
Listing 9.2 attention.ipynb
# In[]:
torch.manual_seed(0)
x = torch.rand(2, 3)
query = F.linear(x, weight=torch.rand(3, 3),
bias=torch.rand(3)) #1
key = F.linear(x, weight=torch.rand(3, 3), bias=torch.rand(3))
value = F.linear(x, weight=torch.rand(3, 3), bias=torch.rand(3))
def dot_product_attention_single(q, k, v):
attn_weights = q @ k.T #2
attn_weights = F.softmax(attn_weights, dim=-1) #3
output = attn_weights @ v #4
return output
dot_product_attention_single(query, key, value)
# Out[]:
tensor([[1.3667, 0.6913, 1.0614],
[1.3718, 0.6930, 1.0627]])
Inspect the runnable attention listing to see value-vector creation and the final weighted combination in PyTorch. section 9.5.1
