liveBook · Manning

Mixture of experts

As explained in Build a DeepSeek Model (From Scratch)

DeepSeek-MoE is a mixture-of-experts feed-forward architecture that routes inputs among specialized expert networks.

A mixture of experts (MoE) replaces a Transformer’s dense feed-forward network with multiple smaller expert networks. A router selects only a subset for each token, allowing total parameter count to grow without a proportional increase in per-token computation. This efficiency comes with routing complexity and the risk that some experts receive much more work than others. section 4.1.2 section 1.2.1

Figure 4.3 The architectural change of MoE. The single, dense FFN is replaced by a collection of four smaller, specialized expert networks.

Compare a single dense feed-forward network with four parallel experts. section 4.1.2

A sparse MoE forward pass proceeds token by token conceptually:

  • Score the experts with a learnable linear router and select the top k for each token. section 4.7
  • Apply softmax to the selected scores, run the token through the selected experts, and form a weighted sum of their outputs. section 4.7 section 4.2.5
  • Stack the resulting vectors to recover an output matrix with the input’s shape. Practical GPU implementations can dispatch and combine tokens using vectorized scatter/gather operations. section 4.2.5

Combining expert outputs

For four tokens with eight-dimensional embeddings, the input and final output both have shape (4, 8). If a token selects experts E1 and E3 with weights 0.9 and 0.1, its output is 0.9 × E1(token) + 0.1 × E3(token). section 4.2.1 section 4.2.5

DeepSeekMoE adds fine-grained expert segmentation and shared expert isolation. Smaller routed experts can specialize more narrowly, while shared experts process every token and can learn knowledge needed across domains. The layer combines shared outputs with the weighted outputs of selected routed experts; a residual connection may be added at the layer or enclosing block. section 4.7 section 4.4.3 section 6.2.1

Figure 4.27 The complete forward pass of the DeepSeekMoE module, showing the three parallel data paths: the dense shared expert path, the sparse routed expert path with dynamic load balancing, and the residual connection.

Follow the shared, routed, and residual paths through the illustrated DeepSeekMoE module. section 4.5

Load imbalance can leave popular experts overloaded and other experts underused. Instead of adding an auxiliary balancing loss, the DeepSeek-style implementation adjusts a per-expert router bias from observed assignment counts. Underloaded experts receive a positive adjustment and overloaded experts a negative one; the implementation bounds the update with tanh. section 4.7 section 4.5

Listing 4.5 The update_bias Method for Load Balancing
@torch.no_grad()
def update_bias(self, x: torch.Tensor):
    # Call once per optimizer step on the same tokens seen by forward.
    # Uses the SAME logits (including current bias) to estimate loads.
    
    N = x.shape[0] * x.shape[1]
    logits = F.linear(x.reshape(-1,
        self.d_model), self.centroids) + self.bias  #A
    
    _, idx = torch.topk(logits, self.top_k, dim=-1)
 
    counts = torch.bincount(idx.flatten(),
        minlength=self.n_routed).float()  #B
    
    avg = counts.sum() / max(1, self.n_routed)
 
    # Smooth, bounded update; avoids large jumps from sign()
    # violation > 0 => under-loaded (we want to increase its prior)
    violation = (avg - counts) / (avg + 1e-6)  #C
 
    self.bias.add_(self.bias_lr * torch.tanh(violation))  #D

Inspect the count-based bias update used once per optimizer step in the standalone module. section 4.5

In one TinyStories experiment, models with nearly equal parameter counts were trained for 5,000 iterations. The standard MoE reached a best validation loss of 1.9854 in 14.29 minutes; DeepSeek-MoE reached 1.9451 in 11.67 minutes. These results describe that particular small-model comparison, not a universal performance guarantee. section 4.6

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Works through routing mathematics, expert dispatch, bias updates, and a complete PyTorch module, while noting vectorized production dispatch.
Examples
Traces four eight-dimensional token embeddings through three experts and compares standard and DeepSeek-style MoE on TinyStories.
What it assumes
Builds from first principles, beginning with the standard Transformer feed-forward network before introducing sparse experts.
What you can do after
Build a DeepSeek-style MoE module from scratch, including shared experts, sparse routing, and dynamic load balancing.
How it connects
Connects MoE to feed-forward networks, sparsity, softmax routing, residual connections, and auxiliary-loss load balancing.

Also explained in

Compare how 10 books explain mixture of experts →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.