A query is the representation of what a position in a sequence is looking for. In self-attention, each input position supplies a query for its own output. Comparing that query with the sequence’s keys produces relevance scores; the corresponding values supply the content combined into the output. section 2.2.3
A simplified scaled dot-product attention calculation follows these steps:
- Project the input embeddings separately into query, key, and value matrices: Q, K, and V. section 2.2.3
- Compute query–key scores with Q × Kᵀ, scale them by the square root of the key dimension, and apply softmax to obtain attention weights. section 2.2.3
- Multiply the weights by V to produce outputs that combine information from the sequence. section 2.2.3
“The movie was not bad”
Treat “not” as the query. The keys indicate how relevant the sentence’s words are to it, so “bad” may receive a high weight. The values provide the content that the weighted result incorporates. section 2.2.3
Listing 2.3 Simplified scaled dot-attention implementation
embed_1 = np.array([0, 1, 0]) #1 embed_2 = np.array([1, 0, 1]) embed_3 = np.array([0, 1, 1]) embed_4 = np.array([1, 1, 0]) embeddings = np.array([embed_1, embed_2, embed_3, embed_4]) #2 Wq = rand(3, 3) #3 Wk = rand(3, 3) Wv = rand(3, 3) Q = embeddings.dot(Wq) #4 K = embeddings.dot(Wk) V = embeddings.dot(Wv) attention_scores = softmax(Q.dot(K.T) / sqrt(K.shape[1]), axis=1) #5 attention_output = attention_scores.dot(V) #6
The NumPy listing makes the distinction concrete: Wq creates Q, while Wk and Wv create K and V; Q participates in scoring, and V supplies the weighted output. section 2.2.3
