A shared expert is a feed-forward subnetwork in a mixture-of-experts (MoE) layer that processes every input token, independently of the router. It provides a dense path alongside the sparse path of selected routed experts. section 4.4.3
The motivation is to reduce duplicated knowledge. For example, experts specializing in code and legal contracts may both need English grammar. Shared experts are intended to carry common knowledge, leaving routed experts more capacity for specialization. These domain labels are intuition, not preset expert assignments: specialization emerges during training. section 4.4.3 section 4.1.3
Figure 4.21 The DeepSeekMoE architecture with Shared and Routed Experts. All tokens are processed by the dense Shared Experts, while the router selectively sends each token to a sparse subset of the Routed Experts.

Follow each token through the dense shared path and the router-selected sparse path. section 4.4.3
For token x, combine the paths as: output = residual(x) + sum(shared expert outputs) + weighted sum(selected routed expert outputs). The router selects the top-k routed experts; it does not select whether shared experts run. section 4.4.3
A practical shared expert can be a two-layer feed-forward network: project to a hidden dimension, apply GELU and dropout, then project back to the model dimension. In the forward pass, initialize shared_out with zeros shaped like x and add each shared expert’s output. For an input shaped [batch, sequence, model dimension], this preserves the input shape for the final sum. section 4.5
Listing 4.3 The Shared Expert Path in the forward Method
def forward(self, x: torch.Tensor) -> torch.Tensor:
B, S, D = x.shape
x_flat = x.reshape(-1, D) # [N, D] with N=B*S
# 1) shared path
shared_out = torch.zeros_like(x)
for exp in self.shared:
shared_out += exp(x)
# ... (routed path comes next)The shared-path loop applies every shared expert to the full input and accumulates its output. section 4.5
Matched-parameter comparison
On TinyStories, a DeepSeek-MoE model with shared experts and dynamic balancing achieved best validation loss 1.9451, versus 1.9854 for a standard MoE, with about 101.3 million parameters each. The comparison changes more than shared experts, so it does not isolate their contribution. section 4.6
