A query represents what the model is looking for at one position in a sequence. Attention compares each query with keys from other positions, then retrieves values from the positions that best match. This lets the model focus on contextually relevant information instead of treating every position equally. section 9.5
In dot-product self-attention, queries participate in this sequence:
- Project each input position into a query vector using its own learned weights and bias. section 9.5.1
- Compute dot products between query and key vectors to score how relevant each position is. section 9.5.1
- Apply softmax to normalize the scores into attention weights, then use those weights to combine value vectors. section 9.5.1
Query projection in PyTorch
For an input x with shape [2, 3], a query matrix can be created with `F.linear(x, weight, bias)`. The resulting queries can then be compared with keys using `q @ k.T`. section 9.5.1
Listing 9.2 attention.ipynb
# In[]:
torch.manual_seed(0)
x = torch.rand(2, 3)
query = F.linear(x, weight=torch.rand(3, 3),
bias=torch.rand(3)) #1
key = F.linear(x, weight=torch.rand(3, 3), bias=torch.rand(3))
value = F.linear(x, weight=torch.rand(3, 3), bias=torch.rand(3))
def dot_product_attention_single(q, k, v):
attn_weights = q @ k.T #2
attn_weights = F.softmax(attn_weights, dim=-1) #3
output = attn_weights @ v #4
return output
dot_product_attention_single(query, key, value)
# Out[]:
tensor([[1.3667, 0.6913, 1.0614],
[1.3718, 0.6930, 1.0627]])
The listing shows query, key, and value projections followed by dot-product scoring, softmax normalization, and value aggregation. section 9.5.1
