A routed expert is a specialized feed-forward network selected sparsely by a router. For each token, the router chooses the top-k experts rather than activating the entire expert pool, allowing different parameters to specialize in different learned patterns. Specialization is emergent and may be polysemantic rather than cleanly human-interpretable. section 4.1.3 section 4.4.3
A routed expert is typically a two-layer MLP with GELU, often using a smaller hidden dimension than a dense feed-forward network. During the forward pass, router logits are computed from token representations, optionally in mixed precision; top-k indices are selected, and a softmax converts their logits into gates. Each expert processes only the tokens assigned to it, and the gated outputs are scattered back into the token layout. section 4.5 section 4.5
A DeepSeek-style MoE layer uses routed experts as one part of a three-path computation:
- Route each token to its top-k experts using router logits plus a learnable balancing bias. section 4.5
- Process every token through dense shared experts, which retain common foundational knowledge while routed experts specialize. section 4.4.3 section 4.5
- Combine the residual connection, shared-expert outputs, and gate-weighted routed-expert outputs. section 4.4.3 section 4.5
- Update the routing bias from observed top-k counts so under-loaded experts receive a higher prior without an auxiliary loss. section 4.5
Figure 4.27 The complete forward pass of the DeepSeekMoE module, showing the three parallel data paths: the dense shared expert path, the sparse routed expert path with dynamic load balancing, and the residual connection.

Trace the dense shared path, sparse routed path, dynamic load balancing, and residual connection as one complete forward pass. section 4.5
