Expert routing selects specialized subnetworks for incoming tokens instead of activating one generalist feed-forward network for every input. Learned specialization enables sparsity: irrelevant experts remain dormant, reducing computation. section 4.1.3
The routing process performs three linked operations:
- Inspect each incoming token and select the most appropriate specialists from the available experts. section 4.1.3
- Keep unrelated experts dormant when the input does not require their learned capabilities, saving computation. section 4.1.3
- Combine the selected expert outputs into one matrix with the shape expected by the next Transformer layer. section 4.2.1
Routing “What is 1+1?”
A router may activate a punctuation expert for the question mark and a verb expert for “is,” while leaving noun-related experts dormant. A numbers expert might not receive priority, because routing follows learned patterns rather than only the apparent topic. section 4.1.3
Figure 4.4 An example of expert routing in a Mixture-of-Experts model. For the input "What is 1+1?", the router must decide which specialized experts to activate. The routing mechanism might prioritize grammatical components (like the question mark and the verb "is"), highlighting how routing is a nuanced decision based on learned patterns.

Use the routing diagram to trace how individual tokens can activate different specialists and leave other experts inactive. section 4.1.3
Figure 4.5 The initial challenge of MoE. The input matrix is passed through each of the three expert networks in parallel, resulting in three separate expert output matrices.

Use the matrix diagram to visualize why multiple expert results must be combined into one output matrix before continuing through the Transformer. section 4.2.1


