A decoder layer, also called a decoder block, is a repeatable Transformer unit for processing a sequence while predicting what comes next. Its self-attention is masked so a position can use earlier tokens but not later ones. Stacking layers gives the decoder more processing stages without changing the basic pattern. section 9.3.1 section 11.2.3
The layer has two forms here, depending on what information the decoder receives:
- In an encoder–decoder Transformer, it has three sublayers: masked multihead self-attention, cross-attention to the encoder output, and a feed-forward network. Each sublayer has layer normalization and a residual connection. Six such layers form the decoder. section 9.3.1 section 9.3.2
- In a decoder-only GPT model, there is no encoder output to attend to. Each block instead has causal self-attention and a position-wise feed-forward network; both have layer normalization and a residual connection. GPT-2XL stacks 48 blocks. section 11.1.1 section 11.2.4
Follow one translation input
Suppose the target input is ['BOS', 'comment', 'et', 'es-vous', '?'], embedded as a (1, 5, 256) tensor: one sequence, five tokens, and 256 values per token. A 5 × 5 target mask permits each position to attend to itself and preceding positions. The layer then uses cross-attention to the encoder output before applying its feed-forward network. The decoder is used to predict ['comment', 'et', 'es-vous', '?', 'EOS']. section 9.3.1
Listing 9.4 Creating a decoder layer
class DecoderLayer(nn.Module):
def __init__(self, size, self_attn, src_attn,
feed_forward, dropout):
super().__init__()
self.size = size
self.self_attn = self_attn
self.src_attn = src_attn
self.feed_forward = feed_forward
self.sublayer = nn.ModuleList([deepcopy(
SublayerConnection(size, dropout)) for i in range(3)])
def forward(self, x, memory, src_mask, tgt_mask):
x = self.sublayer[0](x, lambda x:
self.self_attn(x, x, x, tgt_mask)) #1
x = self.sublayer[1](x, lambda x:
self.src_attn(x, memory, memory, src_mask)) #2
output = self.sublayer[2](x, self.feed_forward) #3
return output
The `DecoderLayer` implementation shows the three sublayers in order. `tgt_mask` is used for self-attention; `memory` and `src_mask` are used for cross-attention. section 9.3.1
For a decoder-only implementation, combine `CausalSelfAttention` with a feed-forward network: `x = x + attn(norm(x))`, then `x = x + mlp(norm(x))`. Stack the resulting blocks after token and positional embeddings, then use a final normalization and linear head to produce vocabulary logits. The same layout supports a three-layer text model and a six-layer music model by changing model dimensions and vocabulary. section 11.2.4 section 12.3.3 section 14.3.2
