A linear layer maps an input representation to an output through a learned affine transformation: y = Wx + b, where W is a learned weight matrix and b is an optional bias. Its output dimension can differ from the input dimension, making the layer useful for expanding, compressing, or remapping representations. section 2.2.1 section 4.1.1 section 6.2.2
In a transformer, linear layers serve several distinct jobs:
- Attention projection: concatenate the head outputs, then apply learned weights to mix information across heads. The projection can remap the representation even when its dimension stays unchanged. section 2.2.1
- Feed-forward network: expand each token’s context vector, apply a nonlinear activation, then contract it to the original dimension. A typical expansion factor is four; dense computation uses all FFN parameters for every token. section 2.2.2 section 4.1.1
- Output layer: project final context vectors into vocabulary-sized logits, giving a score for each vocabulary item. section 2.2.2
- Multi-token prediction: concatenate a normalized previous hidden state and future-token embedding, then project the combined 2d vector back to dimension d before a transformer block. section 5.3.2
Equation 5.1
The MTP projection combines two normalized vectors before mapping their concatenation to the transformer input. section 5.3.2
A vocabulary projection in PyTorch
The output head can be declared as nn.Linear(args.d_model, args.vocab_size, bias=False). It maps each final hidden state from model dimension to vocabulary logits without a bias. During inference, computing logits only for the last token avoids unnecessary output computation. section 6.2.2
Figure 4.2 The standard Feed-Forward Network (FFN) in a Transformer block, featuring an expansion-contraction architecture.

The FFN diagram shows how two linear layers expand and then restore the representation dimension. section 4.1.1
















