Cross-attention lets one representation query another: decoder states supply queries, while encoder outputs supply keys and values. It transfers information between sequences instead of relating positions within one sequence alone. section 9.3.1 section 9.1.2
In a Transformer decoder, masked self-attention first uses previously available target tokens. Cross-attention then connects those intermediate states to the encoder output, helping generate each translated token from the input sequence. section 9.1.2 section 9.3.1
The operation proceeds in a small sequence of steps:
- Transform the decoder input into query vectors Q and the encoder output into key vectors K. section 9.3.1
- Compute scaled dot products between Q and K, dividing by the square root of the key dimension. section 9.3.1
- Apply softmax to obtain weights showing how each decoder element relates to encoder elements, then use the encoder values to transfer information. section 9.3.1
Inside a decoder layer
For decoder input ['BOS', 'comment', 'et', 'es-vous', '?'], masked self-attention runs first. The resulting tensor attends to encoder memory through cross-attention, and a feed-forward network produces the layer output. The decoder layer applies residual connections and layer normalization around these sublayers. section 9.3.1 section 9.1.2
Listing 9.4 Creating a decoder layer
class DecoderLayer(nn.Module):
def __init__(self, size, self_attn, src_attn,
feed_forward, dropout):
super().__init__()
self.size = size
self.self_attn = self_attn
self.src_attn = src_attn
self.feed_forward = feed_forward
self.sublayer = nn.ModuleList([deepcopy(
SublayerConnection(size, dropout)) for i in range(3)])
def forward(self, x, memory, src_mask, tgt_mask):
x = self.sublayer[0](x, lambda x:
self.self_attn(x, x, x, tgt_mask)) #1
x = self.sublayer[1](x, lambda x:
self.src_attn(x, memory, memory, src_mask)) #2
output = self.sublayer[2](x, self.feed_forward) #3
return output
Inspect the DecoderLayer implementation to see self-attention use x for queries, keys, and values, while cross-attention uses the intermediate x as queries and encoder memory as keys and values. section 9.3.1
