liveBook · Manning

Routed expert

As explained in Build a DeepSeek Model (From Scratch)

A routed expert is an expert subnetwork that receives tokens selectively through the mixture-of-experts routing mechanism.

A routed expert is a specialized feed-forward network selected sparsely by a router. For each token, the router chooses the top-k experts rather than activating the entire expert pool, allowing different parameters to specialize in different learned patterns. Specialization is emergent and may be polysemantic rather than cleanly human-interpretable. section 4.1.3 section 4.4.3

A routed expert is typically a two-layer MLP with GELU, often using a smaller hidden dimension than a dense feed-forward network. During the forward pass, router logits are computed from token representations, optionally in mixed precision; top-k indices are selected, and a softmax converts their logits into gates. Each expert processes only the tokens assigned to it, and the gated outputs are scattered back into the token layout. section 4.5 section 4.5

A DeepSeek-style MoE layer uses routed experts as one part of a three-path computation:

  • Route each token to its top-k experts using router logits plus a learnable balancing bias. section 4.5
  • Process every token through dense shared experts, which retain common foundational knowledge while routed experts specialize. section 4.4.3 section 4.5
  • Combine the residual connection, shared-expert outputs, and gate-weighted routed-expert outputs. section 4.4.3 section 4.5
  • Update the routing bias from observed top-k counts so under-loaded experts receive a higher prior without an auxiliary loss. section 4.5
Figure 4.27 The complete forward pass of the DeepSeekMoE module, showing the three parallel data paths: the dense shared expert path, the sparse routed expert path with dynamic load balancing, and the residual connection.

Trace the dense shared path, sparse routed path, dynamic load balancing, and residual connection as one complete forward pass. section 4.5

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Implements ExpertFFN construction, top-k gating, token dispatch, output combination, and bias-based load balancing.
Examples
Uses top-k routing illustrations, ExpertFFN PyTorch listings, and a token-count bias update for load balancing.
What it assumes
Assumes familiarity with feed-forward networks, tensors, PyTorch modules, and mixture-of-experts routing while grounding expert specialization.
What you can do after
Build it from scratch as a PyTorch routed-expert component with sparse dispatch, gating, shared paths, and balancing updates.
How it connects
Connects mixture-of-experts, router, shared experts, feed-forward network, load balancing, gating, and residual connection.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.