In a long sequence, not every token matters equally to the current computation. Attention lets a model score which parts of the input are relevant and weight them accordingly. It was introduced as a way to retain strong sequence modeling while avoiding the slow training associated with LSTMs; it later became central to transformers, which dispense with recurrence. section 2.2.7 section 2.2.8 section 2.3
Think of a dictionary lookup with an extra ingredient: a query. Input vectors are transformed into queries, keys, and values. Comparing a query with keys produces relevance scores; those scores determine how strongly the corresponding values contribute. In the illustrated dot-product implementation, matrix multiplication produces the scores and softmax turns them into weights. section 2.2.8
Listing 2.8 Multihead attention implementation
import numpy as np
from scipy.special import softmax
x = np.array([[1.0, 0.0, 1.0, 0.0],
[0.0, 2.0, 0.0, 2.0],
[1.0, 1.0, 1.0, 1.0]]) #1
w_query = np.array([1,0,1],
[1,0,0],
[0,0,1],
[0,1,1]]) #2
w_key = np.array([[0,0,1],
[1,1,0],
[0,1,0],
[1,1,0]])
w_value = np.array([[0,2,0],
[0,3,0],
[1,0,3],
[1,1,0]])
Q = np.matmul(x,w_query) #3
K = np.matmul(x,w_key) #3
V = np.matmul(x,w_value) #3
k_d = 1 #4
attention_scores = (Q @ K.transpose())/k_d #4
attention_scores[0] = softmax(attention_scores[0]) #5
attention_scores[1] = softmax(attention_scores[1]) #5
attention_scores[2] = softmax(attention_scores[2]) #5
attention1 = attention_scores[0].reshape(-1,1) #6
attention1 = attention_scores[0][0]*V[0] #6
attention2 = attention_scores[0][1]*V[1] #6
attention3 = attention_scores[0][2]*V[2] #6
attention_input1 = attention1 + attention2 + attention3 #7
attention_head1 = np.random.random((3,64)) #8
z0h1 = np.random.random((3,64)) #9
z1h2 = np.random.random((3,64)) #9
z2h3 = np.random.random((3,64)) #9
z3h4 = np.random.random((3,64)) #9
z4h5 = np.random.random((3,64)) #9
z5h6 = np.random.random((3,64)) #9
z6h7 = np.random.random((3,64)) #9
z7h8 = np.random.random((3,64)) #9
Output_attention = np.hstack((z0h1,z1h2,z2h3,z3h4,z4h5,z5h6,z6h7,z7h8)) #10
def dot_product_attention(query, key, value, mask, scale=True): #11
assert query.shape[-1] == key.shape[-1] == value.shape[-1], "q,k,v have different dimensions!"
if scale:
depth = query.shape[-1]
else:
depth = 1
dots = np.matmul(query, np.swapaxes(key, -1, -2)) / np.sqrt(depth)
if mask is not None:
dots = np.where(mask, dots, np.full_like(dots, -1e9))
logsumexp = scipy.special.logsumexp(dots, axis=-1, keepdims=True)
dots = np.exp(dots - logsumexp)
attention = np.matmul(dots, value)
return attention
def masked_dot_product_self_attention(q,k,v,scale=True): #12
mask_size = q.shape[-2]
mask = np.tril(np.ones((1, mask_size, mask_size), dtype=np.bool_), k=0)
return DotProductAttention(q,k,v,mask,scale=scale)
The NumPy listing traces input vectors through query, key, and value projections to attention scores and softmax weights. section 2.2.8
The permitted relationships depend on the task:
- Bidirectional self-attention compares positions within the same sequence in both directions. section 2.2.8
- Causal attention restricts focus to preceding positions; masking can also hide parts of a sequence for a model to predict. section 2.2.8
Attention is not a free shortcut. Scoring every pair of tokens scales quadratically with sequence length. A larger token limit can therefore mean slower inference, more timeouts, or hardware memory limits. When an application needs a response within a deadline, a longer context window is useful only if its latency and reliability remain acceptable. section 3.2.6
