An embedding is a learned vector representation of a discrete input such as a token. It carries information used by later neural-network transformations, including attention and expert routing. section 3.2.1 section 4.2.3
Treat the embedding matrix as the model’s numerical interface to a token sequence:
- Represent four tokens as an input matrix X with shape (4, 8), giving each token an 8-dimensional embedding. section 3.2.1 section 2.3.2
- Project X through learned weights to produce model-specific representations such as queries, keys, values, or routing scores. section 3.2.1 section 4.2.3
- Keep the original embedding distinct from a compressed latent vector: MLA obtains cKV only after multiplying X by its KV Down-Projector. section 3.2.2 section 3.4.1
From “bright” to a cached latent
When the token “bright” arrives, its embedding X_bright is multiplied by Wq to form a query and by Wdkv to form a small compressed latent vector. The embedding is therefore the fresh token representation; the latent vector is a downstream, cache-oriented transformation. section 3.4.1 section 3.4.2







































































