liveBook · Manning

Additional head

As explained in Build a DeepSeek Model (From Scratch)

An output head is the final neural-network component that transforms hidden representations into task-specific output scores or predictions.

An additional head extends next-token prediction by forecasting one further future token. In DeepSeek-style multi-token prediction, heads are chained rather than independent: each head predicts a token and produces a refined hidden state for the next head. section 5.3.2

A chained prediction head performs two coupled operations:

  • Receives the previous head's hidden state and the embedding of its target future token. section 5.3.2
  • Normalizes, merges, and projects those inputs before applying a Transformer block that refines the representation. section 5.3.2
  • Projects the refined state through a shared un-embedding matrix to obtain vocabulary logits and select the predicted token. section 5.3.2

Prediction depth D=3

Set up three chained MTP heads. Head 1 predicts the next token, passes its refined state to Head 2, and Head 2 passes its state to Head 3. The number of heads therefore defines the prediction depth. Other implementations may reuse one lightweight head across depths, so the chain is an architectural choice rather than the only design. section 5.3.2

Figure 5.8 The sequential architecture of the DeepSeek MTP modules. The hidden state from one module is passed as an input to the next, forming a causal chain.

Use the sequential architecture diagram to trace how each head passes a refined hidden state into the next prediction step. section 5.3.2

Figure 5.12 The final prediction steps within an MTP module. The new hidden state generated by the MTP Transformer block is passed to a "Shared Un-Embedding Matrix." This projects the hidden state into the vocabulary space to produce a logits vector, from which the final token for that prediction step is selected.

Use the final-step diagram to inspect the projection from a refined hidden state into vocabulary logits. section 5.3.2

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains inputs, normalization, projection, Transformer refinement, and vocabulary projection, while deferring implementation-specific alternatives.
Examples
Uses a three-head chain with prediction depth D=3 and diagrams of sequential flow and final vocabulary projection.
What it assumes
Assumes Transformer hidden states, token embeddings, and vocabulary projection while grounding their roles inside one prediction head.
What you can do after
Build it from scratch as a chained module that predicts future tokens and forwards refined hidden states.
How it connects
Connects hidden states, token embeddings, Transformer blocks, vocabulary logits, shared un-embedding, and multi-token prediction.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.