Multi-head attention lets a transformer consider several relationships between sequence positions at once. Rather than relying on one attention pattern, it gives the model multiple perspectives on the same input. This helps it capture long-range relationships without processing tokens one timestep at a time, as an RNN does. section 1.1.4 section 2.1.2
Start with self-attention: each position receives a weighted representation of positions in its own sequence, including itself. A scaled dot-product calculation projects inputs into queries, keys, and values; computes query–key scores; applies softmax to obtain weights; and uses those weights to combine values. Multi-head attention projects and divides the representations across heads so these calculations can capture different aspects of the input. section 2.2.3
Negation changes sentiment
In “The movie was not bad,” the relationship between “not” and “bad” matters more than either word alone: together they support a positive reading. Multiple attention perspectives help capture such interactions. section 1.1.4
In a transformer encoder layer, multi-head self-attention precedes a feed-forward network. An implementation can project queries, keys, and values, divide the model dimension among heads, and pass the attention output through the layer’s residual connection, dropout, and normalization. An attention mask can also be supplied. section 2.2.1 section 2.2.3
Listing 2.4 Simplified multihead attention implementation
class MultiHeadAttention(nn.Module):
def __init__ (self, d_model, num_heads):
super(MultiHeadAttention, self). __init__ ()
assert d_model % num_heads == 0
self.d_model = d_model
self.num_heads = num_heads
self.d_k = d_model // num_heads
self.W_q = nn.Linear(d_model, d_model)
self.W_k = nn.Linear(d_model, d_model)
self.W_v = nn.Linear(d_model, d_model)
self.W_o = nn.Linear(d_model, d_model)
def forward(self, query, key, value, mask=None):
batch_size = query.size(0)
query = self.W_q(query).view(batch_size, -1,
self.num_heads, self.d_k).transpose(1, 2)
key = self.W_k(key).view(batch_size, -1,
self.num_heads, self.d_k).transpose(1, 2)
value = self.W_v(value).view(batch_size, -1,
self.num_heads, self.d_v).transpose(1, 2)
scores = torch.matmul(query, key.transpose(-2, -1)) /
math.sqrt(self.d_k)
if mask is not None:
scores = scores.masked_fill(mask == 0, -1e9)
attention = F.softmax(scores, dim=-1)
output = torch.matmul(attention, value).transpose(1,
2).contiguous().view(batch_size, -1, self.d_model)
output = self.W_o(output)
return output
Inspect the PyTorch implementation to see the divisibility check, query/key/value projections, and tensor reshaping that separates heads. section 2.2.3
A concrete tensor configuration
With a model dimension of 512 and eight heads, the illustrated implementation assigns 64 dimensions per head. It passes a batch of 32 sequences, each 100 positions long, as the query, key, and value inputs for self-attention. section 2.2.3
