A feedforward network transforms an input through learned weights, biases, and activation functions, moving information forward without recurrent state or feedback connections. In a Transformer, it receives the blended clues produced by self-attention and reshapes relationships among features. A useful analogy is cooking: attention gathers ingredients, while the feedforward network amplifies, suppresses, or combines them into richer features. section 11.24
A small Transformer feedforward block can be traced as follows:
- Project the original 4-dimensional representation up to 8 dimensions with W1 and bias b1. section 11.24
- Apply ReLU to introduce a nonlinear transformation, producing an 8-dimensional hidden vector. section 11.26
- Project the hidden vector back to 4 dimensions with W2 and bias b2, matching the input size. section 11.26
- Add the original input through a residual connection, allowing the block to be bypassed when its transformation is unnecessary. section 11.26
From text label to guidance vector
A diffusion model can average token embeddings for “a happy face,” pass the resulting vector through two learned feedforward projections with SiLU activation, and inject the refined vector into denoising. The same pattern processes the timestep embedding. The feedforward network acts like a translator: it turns a generic text-encoder representation into guidance specialized for the image model. section 12.11 section 12.15
Figure 11.44 The architecture of the feed-forward network to output token probabilities
Use the architecture figure to follow the project-up, activation, project-down, and token-probability pathway. section 11.24
Mixture-of-Experts models replace one large feedforward network with many smaller experts and route each token to only one or two selected networks, increasing capacity without activating every parameter. section 11.41
