A feedforward network is a position-wise transformation that processes every sequence position independently. In a transformer, it follows self-attention and refines each position’s representation without modeling interactions between positions itself. section 2.1.2 section 2.3.1
A typical transformer feedforward sublayer proceeds through these stages:
- Expand each hidden vector with a linear layer. For hidden size 48, an intermediate size of 192 provides the commonly used fourfold expansion. section 3.3.4
- Apply GELU to the expanded representation, using a differentiable activation instead of ReLU in the example implementation. section 3.3.4
- Project the result back to the original hidden size so the sublayer output can be added to its input. section 3.3.4
- Apply layer normalization and a residual connection around the sublayer as part of the encoder or decoder block. section 2.1.2 section 2.3.2
PyTorch-style MLP
The mechanism can be implemented with two linear layers and GELU: dense_1 maps hidden_size to intermediate_size, activation applies GELU, and dense_2 maps intermediate_size back to hidden_size. An encoder block normalizes its input, adds the attention output, normalizes again, applies the MLP, and adds the MLP output.
