liveBook · Manning

Routing matrix

As explained in Build a DeepSeek Model (From Scratch)

A routing matrix is a learned matrix that projects token representations into one routing score for each expert.

A routing matrix is a trainable linear layer that maps token representations to one raw routing score per expert. For an input matrix of shape (4, 8) and three experts, the routing matrix has shape (8, 3), producing an expert selector matrix of shape (4, 3). section 4.2.3

Figure 4.7 The routing mechanism. The input matrix is multiplied by a learned routing matrix to produce an expert selector matrix, which contains a raw score for each expert for each token.

The routing mechanism shows the matrix multiplication from token representations to per-token, per-expert scores. section 4.2.3

Interpret the resulting expert selector matrix as follows:

  • Each row represents one token, and each column represents one expert. section 4.2.3
  • Each entry is a raw, unnormalized suitability score; top-k selection keeps the highest-scoring experts for each token. section 4.2.3 section 4.2.4
  • After masking non-selected scores with negative infinity and applying softmax, the selected scores become weights that sum to one. section 4.2.4

Routing one token

For the token "The," suppose Expert 2 receives weight 0.6 and Expert 3 receives weight 0.4. The model combines those experts' outputs as 0.6 × Output_E2 + 0.4 × Output_E3, while ignoring Expert 1. The same weighted combination runs across all tokens, preferably through vectorized scatter/gather operations rather than token-by-token loops. section 4.2.5

The routing matrix is also adjusted to influence expert utilization. An auxiliary loss can penalize variation in expert importance, measured from selector-weight columns with the coefficient of variation. An alternative maintains per-expert biases, increasing underloaded experts' scores and decreasing overloaded experts' scores before top-k selection. section 4.3.1 section 4.4.4

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains dimensions, score interpretation, top-k weighting, output combination, and balancing, but omits full training code.
Examples
Uses four tokens, eight-dimensional embeddings, three experts, and weighted outputs for the token "The."
What it assumes
Assumes token embeddings and matrix multiplication, while grounding expert routing through explicit tensor shapes and numerical examples.
What you can do after
Build it from scratch by projecting token embeddings, selecting top-k experts, normalizing weights, and combining expert outputs.
How it connects
Connects routing matrices to mixture-of-experts, routers, top-k selection, softmax, auxiliary loss, and load balancing.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.