liveBook · Manning

Merged embedding

As explained in Build a DeepSeek Model (From Scratch)

A merged embedding is a vector formed by concatenating normalized hidden-state and token-embedding representations.

A merged embedding combines a previous hidden state with the future token's embedding. Each vector is RMS-normalized, concatenated into a 2d vector, and projected back to dimension d before entering the MTP Transformer block. section 5.3.2

MTP head input

For head k, concatenate RMSNorm(hᵢᵏ⁻¹) with RMSNorm(Emb(tᵢ₊ₖ)). The projection Mₖ converts this merged vector into the Transformer input hᵢ′ᵏ. section 5.3.2

Equation 5.1
𝒉𝑖’ᵏ = 𝑀ₖ[𝑅𝑀𝑆𝑁𝑜𝑟𝑚(ℎ𝑖ᵏ⁻¹) ; 𝑅𝑀𝑆𝑁𝑜𝑟𝑚(𝐸𝑚𝑏(𝑡ᵢ₊ₖ))],

Use the equation to inspect how normalization, concatenation, and projection form the next MTP Transformer input. section 5.3.2

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Traces normalization, concatenation, projection, and subsequent Transformer processing, but does not specify projection parameters or implementation code.
Examples
Uses hᵢᵏ⁻¹, Emb(tᵢ₊ₖ), a 2d concatenation, and projection Mₖ as the concrete construction.
What you can do after
Evaluate/understand how an MTP head prepares contextual and token representations for its dedicated Transformer block.
How it connects
Connects hidden states, token embeddings, RMS normalization, linear projection, Transformer blocks, and multi-token prediction.

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.