liveBook · Manning

Feedforward network

As explained in Build a DeepSeek Model (From Scratch)

A feed-forward neural network transforms its inputs through layers whose connections do not form recurrent cycles.

A feed-forward network transforms inputs through layers whose connections do not form recurrent cycles. In a Transformer, it processes each token's contextualized vector independently after self-attention, adding nonlinear per-token computation. section 1.2.1 section 2.2.2

A standard Transformer feed-forward network follows an expansion-contraction path:

  • Apply a first linear layer that expands the embedding dimension, commonly by a factor of four. section 4.1.1
  • Pass the expanded representation through a nonlinear activation function to support more complex calculations. section 2.2.2
  • Apply a second linear layer that contracts the representation back to its input size, preserving the context matrix shape. section 2.2.2 section 4.1.1
  • Return the result through the Transformer block's residual and normalization pathway before the next block. section 2.2.2

The dense design activates every parameter for every token. This gives the expanded hidden space substantial modeling capacity, but makes feed-forward layers a major training and inference cost; Mixture-of-Experts replaces one dense network with sparsely selected specialized experts. section 4.1.1 section 4.1

From context vectors to logits

For the input “The next day is,” attention first produces a context matrix. The feed-forward network transforms each row independently while retaining the matrix shape; repeated Transformer blocks then prepare the final context vectors for projection into vocabulary logits. section 2.2.2

Figure 4.2 The standard Feed-Forward Network (FFN) in a Transformer block, featuring an expansion-contraction architecture.

Use the expansion-contraction diagram to trace how the feed-forward network widens each token representation and contracts it back to the model dimension. section 4.1.1

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains two-layer expansion, nonlinear transformation, contraction, shape preservation, and dense computational cost; defers parameter-level implementation.
Examples
Uses the context matrix for “The next day is,” GPT-2 block repetition, and the standard expansion-contraction diagram.
What it assumes
Assumes familiarity with token embeddings and attention while grounding the FFN's position and tensor-shape behavior.
What you can do after
Apply the expansion-contraction pattern and evaluate its capacity against its dense training and inference costs.
How it connects
Connects feed-forward networks to self-attention, residual connections, layer normalization, Transformer blocks, and Mixture-of-Experts.

Also explained in

Compare how 20 books explain feedforward network →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.