liveBook · Manning

Linear layer

As explained in Build a DeepSeek Model (From Scratch)

A linear layer is a neural-network layer that maps input representations to outputs through a learned affine transformation.

A linear layer maps an input representation to an output through a learned affine transformation: y = Wx + b, where W is a learned weight matrix and b is an optional bias. Its output dimension can differ from the input dimension, making the layer useful for expanding, compressing, or remapping representations. section 2.2.1 section 4.1.1 section 6.2.2

In a transformer, linear layers serve several distinct jobs:

  • Attention projection: concatenate the head outputs, then apply learned weights to mix information across heads. The projection can remap the representation even when its dimension stays unchanged. section 2.2.1
  • Feed-forward network: expand each token’s context vector, apply a nonlinear activation, then contract it to the original dimension. A typical expansion factor is four; dense computation uses all FFN parameters for every token. section 2.2.2 section 4.1.1
  • Output layer: project final context vectors into vocabulary-sized logits, giving a score for each vocabulary item. section 2.2.2
  • Multi-token prediction: concatenate a normalized previous hidden state and future-token embedding, then project the combined 2d vector back to dimension d before a transformer block. section 5.3.2
Equation 5.1
𝒉𝑖’ᵏ = 𝑀ₖ[𝑅𝑀𝑆𝑁𝑜𝑟𝑚(ℎ𝑖ᵏ⁻¹) ; 𝑅𝑀𝑆𝑁𝑜𝑟𝑚(𝐸𝑚𝑏(𝑡ᵢ₊ₖ))],

The MTP projection combines two normalized vectors before mapping their concatenation to the transformer input. section 5.3.2

A vocabulary projection in PyTorch

The output head can be declared as nn.Linear(args.d_model, args.vocab_size, bias=False). It maps each final hidden state from model dimension to vocabulary logits without a bias. During inference, computing logits only for the last token avoids unnecessary output computation. section 6.2.2

Figure 4.2 The standard Feed-Forward Network (FFN) in a Transformer block, featuring an expansion-contraction architecture.

The FFN diagram shows how two linear layers expand and then restore the representation dimension. section 4.1.1

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains projection dimensions and architectural roles, including FFN expansion and contraction, without detailing how the weights are trained.
Examples
Shows a bias-free PyTorch vocabulary head and a 2d-to-d projection of concatenated inputs in multi-token prediction.
What you can do after
Evaluate/understand how linear projections change or preserve dimensions and combine representations across transformer components.
How it connects
Connects linear layers to multi-head attention, feed-forward networks, normalization, logits, and multi-token prediction.

Also explained in

Compare how 17 books explain linear layer →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.