Scaled dot-product attention produces a weighted representation of a sequence. Each input is projected into a query, key, and value. Query–key dot products measure how strongly positions relate; softmax turns the scores into weights, which are used to sum the values. section 2.2.3
Before softmax, divide the dot products by the square root of the key dimension. Scaling controls score variance and helps stabilize training. section 2.2.3
One output from five inputs
To form the third output, compare the third input with all five inputs, normalize the five scores with softmax, then take their weighted sum. section 2.2.3
Listing 2.3 Simplified scaled dot-attention implementation
embed_1 = np.array([0, 1, 0]) #1 embed_2 = np.array([1, 0, 1]) embed_3 = np.array([0, 1, 1]) embed_4 = np.array([1, 1, 0]) embeddings = np.array([embed_1, embed_2, embed_3, embed_4]) #2 Wq = rand(3, 3) #3 Wk = rand(3, 3) Wv = rand(3, 3) Q = embeddings.dot(Wq) #4 K = embeddings.dot(Wk) V = embeddings.dot(Wv) attention_scores = softmax(Q.dot(K.T) / sqrt(K.shape[1]), axis=1) #5 attention_output = attention_scores.dot(V) #6
A NumPy listing shows the projections, scaled score calculation, softmax, and final multiplication by values. section 2.2.3
