Expert specialization divides a model’s processing capacity among specialist networks instead of forcing one giant generalist feed-forward network to handle every pattern. Sparse routing makes this practical: only a few experts activate for each token, while the rest remain dormant. section 4.1.3
During pre-training, experts learn to handle different kinds of information or tasks. This creates a committee of specialists whose distinct capabilities let the model expand capacity without applying every parameter to every token. section 4.1.3
Routing “What is 1+1?”
A router can send the question mark to a punctuation expert and the verb “is” to a verb expert. Noun and proper-noun experts remain dormant because the input does not require their specializations. section 4.1.3
Traditional expert designs limit specialization through two forms of inefficiency:
- Hybridity mixes unrelated capabilities within experts, preventing each module from becoming narrowly specialized. section 4.4.1
- Redundancy makes multiple experts relearn shared knowledge such as English grammar, consuming capacity that could support deeper domain expertise. section 4.4.1
- DeepSeekMoE addresses these limitations through fine-grained expert segmentation, shared expert isolation, and auxiliary-loss-free load balancing. section 4.4
Figure 4.4 An example of expert routing in a Mixture-of-Experts model. For the input "What is 1+1?", the router must decide which specialized experts to activate. The routing mechanism might prioritize grammatical components (like the question mark and the verb "is"), highlighting how routing is a nuanced decision based on learned patterns.

Trace token-level routing decisions and see how selected specialists remain active while irrelevant experts stay dormant. section 4.1.3
