Attention lets a transformer give different parts of a sequence different weights when forming a representation. Rather than carry one recurrent state through the sequence item by item, self-attention lets each position draw information directly from other positions, including itself. section 1.1.3 section 2.2.3 section 1.1.2
Translation depends on context
When translating “I don’t speak French” into “Je ne parle pas français,” the representation of “speak” must account for nearby words such as “don’t.” Attention scores express how strongly positions relate, helping the encoder capture context for the decoder’s output. section 1.1.3 section 1.1.2
Scaled dot-product attention makes those weights concrete. Project input embeddings into queries (Q), keys (K), and values (V). Compare each query with the keys using dot products, divide by the square root of the key dimension, and apply softmax so each position’s weights sum to one. Multiplying those weights by V produces a context-dependent output for each position: softmax(QKᵀ / √dₖ)V. section 2.2.3
Listing 2.3 Simplified scaled dot-attention implementation
embed_1 = np.array([0, 1, 0]) #1 embed_2 = np.array([1, 0, 1]) embed_3 = np.array([0, 1, 1]) embed_4 = np.array([1, 1, 0]) embeddings = np.array([embed_1, embed_2, embed_3, embed_4]) #2 Wq = rand(3, 3) #3 Wk = rand(3, 3) Wv = rand(3, 3) Q = embeddings.dot(Wq) #4 K = embeddings.dot(Wk) V = embeddings.dot(Wv) attention_scores = softmax(Q.dot(K.T) / sqrt(K.shape[1]), axis=1) #5 attention_output = attention_scores.dot(V) #6
The simplified NumPy listing shows the Q, K, and V projections, scaled score calculation, and weighted output as separate operations. section 2.2.3
Multihead attention applies attention through multiple heads so the model can capture different relationships in parallel. Within a transformer, attention works alongside feed-forward networks; positional encoding supplies information about token order. Direct connections across positions aid long-range dependency modeling and parallel processing compared with sequential RNNs, but self-attention’s cost grows quadratically with sequence length. section 2.1.2 section 2.3 section 9.5
