A feedforward network passes an input through layers toward an output. In a Transformer, it is the fully connected sublayer that follows attention. Attention brings information from different token positions together; the feedforward sublayer then transforms each position’s vector independently, using the same network at every position. section 9.2.1 section 12.3.3
To build a position-wise sublayer:
- Start with a vector of size d_model at each token position. A first linear layer expands it to d_ff; a second projects it back to d_model. A common choice here is d_ff = 4 × d_model, so 256 becomes 1,024 and then 256 again. section 9.2.1
- In the GPT decoder block, place GELU between the projections and apply dropout afterward. The activation adds nonlinearity, helping the block represent complex relationships rather than only chaining linear operations. section 11.2.4 section 12.3.3
- Wrap the sublayer with layer normalization and a residual connection. Transformer encoder layers pair it with self-attention; translation decoder layers put it after self-attention and cross-attention. section 9.1.2 section 9.3.1
Listing 11.5 Constructing a decoder block
class Block(nn.Module):
def __init__(self, config): #1
super().__init__()
self.ln_1 = nn.LayerNorm(config.n_embd)
self.attn = CausalSelfAttention(config)
self.ln_2 = nn.LayerNorm(config.n_embd)
self.mlp = nn.ModuleDict(dict(
c_fc = nn.Linear(config.n_embd, 4 * config.n_embd),
c_proj = nn.Linear(4 * config.n_embd, config.n_embd),
act = GELU(),
dropout = nn.Dropout(config.resid_pdrop),
))
m = self.mlp
self.mlpf=lambda x:m.dropout(m.c_proj(m.act(m.c_fc(x))))
def forward(self, x):
x = x + self.attn(self.ln_1(x)) #2
x = x + self.mlpf(self.ln_2(x)) #3
return x
Inspect the GPT decoder block: c_fc expands the embedding fourfold, GELU activates it, c_proj restores its width, and the result is added back to the block input. section 11.2.4
From text to music
A three-block text generator uses 256-wide embeddings and 1,024-wide feedforward hidden layers before projecting to a 10,600-token vocabulary. A six-block music Transformer applies the same decoder-block pattern to musical event tokens, with 512-wide embeddings, 2,048-wide hidden layers, and a 390-token vocabulary. The feedforward network changes features at each position; the final linear head produces vocabulary logits. section 12.3.3 section 14.3.2
Position-wise processing is the key boundary: the feedforward sublayer does not itself exchange information between tokens. Its input can already contain context gathered by attention, which it then transforms before the next block. section 9.2.1 section 12.3.3
