Knowledge redundancy occurs when multiple MoE experts independently store the same foundational information. Verb- and noun-oriented experts, or Python and medical specialists, may all relearn English grammar instead of using their capacity for narrower expertise. section 4.4.1
Redundancy creates an architectural efficiency problem:
- Common knowledge consumes part of every expert's parameter capacity. section 4.4.1
- Specialized experts spend capacity relearning foundational grammar, limiting deeper specialization. section 4.4.1
- Shared Expert Isolation assigns common knowledge to dense shared experts while routed experts focus on narrow domains. section 4.4.3
In Shared Expert Isolation, every token passes through the dense shared experts, while the router sends each token only to a sparse top-k subset of routed experts. The layer sums both paths: shared outputs provide general knowledge, and weighted routed outputs provide specialization. section 4.4.3
Python and medical-language specialists
A Python specialist and a medical-research specialist require the same subject-verb agreement knowledge to interpret English comments or prose. Shared experts learn that grammar once; routed experts can concentrate on Python syntax or medical content. section 4.4.1 section 4.4.3
Figure 4.21 The DeepSeekMoE architecture with Shared and Routed Experts. All tokens are processed by the dense Shared Experts, while the router selectively sends each token to a sparse subset of the Routed Experts.

Use the architecture figure to trace all-token processing through dense shared experts and selective routing through sparse routed experts. section 4.4.3
