Shared expert isolation separates common knowledge from specialized knowledge in a mixture-of-experts layer. Dense shared experts process every token, while sparse routed experts receive only the top-k assignments from the router. This prevents specialists from repeatedly learning foundational information and preserves their capacity for narrower domains. section 4.4.3
The layer divides its computation into two paths:
- Shared experts act as dense generalists: every token passes through all of them, allowing them to absorb common grammar, facts, and reasoning structures. section 4.4.3
- Routed experts act as sparse specialists: the router selects the top-k experts for each token, concentrating their capacity on unique domains. section 4.4.3
- The layer adds the shared-expert output to the weighted outputs of the selected routed experts. section 4.4.3
Why isolation helps
An expert focused on Python syntax and one focused on legal contracts both need English grammar. Dense shared experts can retain that common knowledge, allowing the routed experts to specialize more completely in programming or law. section 4.4.3
