liveBook · Manning

Embedding

As explained in Build a DeepSeek Model (From Scratch)

An embedding is a learned vector representation of a discrete input such as a token, including its semantic information.

An embedding is a learned vector representation of a discrete input such as a token. It carries information used by later neural-network transformations, including attention and expert routing. section 3.2.1 section 4.2.3

Treat the embedding matrix as the model’s numerical interface to a token sequence:

  • Represent four tokens as an input matrix X with shape (4, 8), giving each token an 8-dimensional embedding. section 3.2.1 section 2.3.2
  • Project X through learned weights to produce model-specific representations such as queries, keys, values, or routing scores. section 3.2.1 section 4.2.3
  • Keep the original embedding distinct from a compressed latent vector: MLA obtains cKV only after multiplying X by its KV Down-Projector. section 3.2.2 section 3.4.1

From “bright” to a cached latent

When the token “bright” arrives, its embedding X_bright is multiplied by Wq to form a query and by Wdkv to form a small compressed latent vector. The embedding is therefore the fresh token representation; the latent vector is a downstream, cache-oriented transformation. section 3.4.1 section 3.4.2

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Defines embeddings through shapes and matrix multiplications, then stops short of explaining how embedding vectors are learned.
Examples
Uses four-token matrices, the token “bright,” and explicit shapes such as (4, 8) to trace embedding transformations.
What you can do after
Apply via matrix multiplication to transform token embeddings into attention inputs, compressed cache entries, or expert scores.
How it connects
Connects embeddings to attention, key-value caching, latent vectors, positional encoding, and mixture-of-experts routing.

Also explained in

Compare how 72 books explain embedding →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.