liveBook · Manning

Expert routing

As explained in Build a DeepSeek Model (From Scratch)

Routing is the mechanism that selects which expert subnetworks process a given input in a mixture-of-experts architecture.

Expert routing selects specialized subnetworks for incoming tokens instead of activating one generalist feed-forward network for every input. Learned specialization enables sparsity: irrelevant experts remain dormant, reducing computation. section 4.1.3

The routing process performs three linked operations:

  • Inspect each incoming token and select the most appropriate specialists from the available experts. section 4.1.3
  • Keep unrelated experts dormant when the input does not require their learned capabilities, saving computation. section 4.1.3
  • Combine the selected expert outputs into one matrix with the shape expected by the next Transformer layer. section 4.2.1

Routing “What is 1+1?”

A router may activate a punctuation expert for the question mark and a verb expert for “is,” while leaving noun-related experts dormant. A numbers expert might not receive priority, because routing follows learned patterns rather than only the apparent topic. section 4.1.3

Figure 4.4 An example of expert routing in a Mixture-of-Experts model. For the input "What is 1+1?", the router must decide which specialized experts to activate. The routing mechanism might prioritize grammatical components (like the question mark and the verb "is"), highlighting how routing is a nuanced decision based on learned patterns.

Use the routing diagram to trace how individual tokens can activate different specialists and leave other experts inactive. section 4.1.3

Figure 4.5 The initial challenge of MoE. The input matrix is passed through each of the three expert networks in parallel, resulting in three separate expert output matrices.

Use the matrix diagram to visualize why multiple expert results must be combined into one output matrix before continuing through the Transformer. section 4.2.1

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Traces routing from token selection through expert outputs and matrix combination, without detailing router equations or training procedures.
Examples
Uses “What is 1+1?” and “The next day is.” alongside token-routing and expert-output matrix diagrams.
What it assumes
Assumes familiarity with tokens, expert feed-forward networks, matrices, and Transformer layers.
What you can do after
Evaluate how selective expert activation produces one Transformer-compatible output while avoiding unnecessary expert computation.
How it connects
Connects mixture-of-experts, sparsity, feed-forward networks, and Transformer layers.

Also explained in

Compare how 3 books explain expert routing →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.