A value is the content an attention mechanism weighs and combines to produce an output vector. Each input vector has three roles: its query looks for relevant inputs, its key helps determine relevance, and its value supplies content to the weighted sum. Keys help set the attention weights; values are multiplied by those weights after softmax. section 2.2.3
In scaled dot-product attention, input embeddings are multiplied by separate matrices to form Q, K, and V. Softmax converts scores computed from Q and K into attention weights; multiplying those weights by V produces the attention output. Values may start from input embeddings, while stacked attention layers develop more contextual representations. section 2.2.3
“The movie was not bad”
If “not” is the query, “bad” may receive a high attention weight because the words are closely related in the phrase. The values for “not” and “bad,” each multiplied by its weight, contribute to the output for “not.” section 2.2.3
Listing 2.3 Simplified scaled dot-attention implementation
embed_1 = np.array([0, 1, 0]) #1 embed_2 = np.array([1, 0, 1]) embed_3 = np.array([0, 1, 1]) embed_4 = np.array([1, 1, 0]) embeddings = np.array([embed_1, embed_2, embed_3, embed_4]) #2 Wq = rand(3, 3) #3 Wk = rand(3, 3) Wv = rand(3, 3) Q = embeddings.dot(Wq) #4 K = embeddings.dot(Wk) V = embeddings.dot(Wv) attention_scores = softmax(Q.dot(K.T) / sqrt(K.shape[1]), axis=1) #5 attention_output = attention_scores.dot(V) #6
The NumPy listing forms V with `embeddings.dot(Wv)`, then computes `attention_scores.dot(V)`. With four three-component embeddings, this produces one three-component output per input. section 2.2.3
