A merged embedding combines a previous hidden state with the future token's embedding. Each vector is RMS-normalized, concatenated into a 2d vector, and projected back to dimension d before entering the MTP Transformer block. section 5.3.2
MTP head input
For head k, concatenate RMSNorm(hᵢᵏ⁻¹) with RMSNorm(Emb(tᵢ₊ₖ)). The projection Mₖ converts this merged vector into the Transformer input hᵢ′ᵏ. section 5.3.2
Equation 5.1
Use the equation to inspect how normalization, concatenation, and projection form the next MTP Transformer input. section 5.3.2
