liveBook · Manning

Un-embedding matrix

As explained in Build a DeepSeek Model (From Scratch)

An un-embedding matrix is a learned output projection that maps hidden representations to scores over a model's vocabulary.

An un-embedding matrix is a learned output projection that maps a hidden state into the model’s full vocabulary space, producing logits for token prediction. section 5.3.2

Within sequential multi-token prediction, the matrix:

  • Receives the refined hidden state produced by an MTP module. section 5.3.2
  • Produces a logits vector covering the complete vocabulary. section 5.3.2
  • Is shared by the main model and all MTP heads. section 5.3.2

Sequential MTP prediction

After an MTP Transformer block creates a new hidden state, the shared un-embedding matrix converts it into vocabulary logits, from which the predicted future token is selected. section 5.3.2

Figure 5.12 The final prediction steps within an MTP module. The new hidden state generated by the MTP Transformer block is passed to a "Shared Un-Embedding Matrix." This projects the hidden state into the vocabulary space to produce a logits vector, from which the final token for that prediction step is selected.

Trace the final prediction path from an MTP module’s hidden state through the shared projection to vocabulary logits and token selection. section 5.3.2

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains the projection’s input, vocabulary-sized logits output, and sharing across prediction heads, without detailing matrix dimensions or implementation code.
Examples
Uses DeepSeek’s sequential MTP architecture and its shared final prediction path as the concrete example.
What you can do after
Apply it to project MTP hidden states into vocabulary logits and select predicted future tokens.
How it connects
Connects hidden states, logits, vocabulary prediction, Transformer blocks, and multi-token prediction modules.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.