A feed-forward network transforms inputs through layers whose connections do not form recurrent cycles. In a Transformer, it processes each token's contextualized vector independently after self-attention, adding nonlinear per-token computation. section 1.2.1 section 2.2.2
A standard Transformer feed-forward network follows an expansion-contraction path:
- Apply a first linear layer that expands the embedding dimension, commonly by a factor of four. section 4.1.1
- Pass the expanded representation through a nonlinear activation function to support more complex calculations. section 2.2.2
- Apply a second linear layer that contracts the representation back to its input size, preserving the context matrix shape. section 2.2.2 section 4.1.1
- Return the result through the Transformer block's residual and normalization pathway before the next block. section 2.2.2
The dense design activates every parameter for every token. This gives the expanded hidden space substantial modeling capacity, but makes feed-forward layers a major training and inference cost; Mixture-of-Experts replaces one dense network with sparsely selected specialized experts. section 4.1.1 section 4.1
From context vectors to logits
For the input “The next day is,” attention first produces a context matrix. The feed-forward network transforms each row independently while retaining the matrix shape; repeated Transformer blocks then prepare the final context vectors for projection into vocabulary logits. section 2.2.2
Figure 4.2 The standard Feed-Forward Network (FFN) in a Transformer block, featuring an expansion-contraction architecture.

Use the expansion-contraction diagram to trace how the feed-forward network widens each token representation and contracts it back to the model dimension. section 4.1.1



















