Knowledge hybridity occurs when too few MoE experts must absorb unrelated knowledge and skills. Instead of specializing, each expert becomes a generalist whose parameters are pulled toward conflicting domains. section 4.4.1 section 4.4.2
One expert, three domains
An expert in an eight-expert model may process Python syntax, legal language, and historical facts. Its parameters become a compromise rather than deeply specialized knowledge for any one domain. section 4.4.1
Fine-grained expert segmentation addresses the problem by replacing a few large experts with many smaller ones. A larger routing pool can specialize tokens more precisely without necessarily increasing total expert capacity or activated computation. section 4.4.2
The capacity trade-off works by dividing existing expert capacity:
- A conventional design with 16 experts of hidden dimension 4096 has the same total capacity as 64 experts of hidden dimension 1024. section 4.4.2
- The wider pool gives routing more specialized destinations while preserving total parameters and potentially top-k activated parameters. section 4.4.2
- Shared expert isolation complements segmentation by assigning common knowledge to dense generalist experts, leaving routed experts freer to specialize. section 4.7
Figure 4.20 A comparison between a conventional MoE with a few large experts (top) and DeepSeek's fine-grained approach with many smaller experts (bottom).

Compare the few-large-expert design with the many-small-expert design to see how segmentation redistributes capacity. section 4.4.2
