A mixture of experts (MoE) replaces a Transformer’s dense feed-forward network with multiple smaller expert networks. A router selects only a subset for each token, allowing total parameter count to grow without a proportional increase in per-token computation. This efficiency comes with routing complexity and the risk that some experts receive much more work than others. section 4.1.2 section 1.2.1
Figure 4.3 The architectural change of MoE. The single, dense FFN is replaced by a collection of four smaller, specialized expert networks.

Compare a single dense feed-forward network with four parallel experts. section 4.1.2
A sparse MoE forward pass proceeds token by token conceptually:
- Score the experts with a learnable linear router and select the top k for each token. section 4.7
- Apply softmax to the selected scores, run the token through the selected experts, and form a weighted sum of their outputs. section 4.7 section 4.2.5
- Stack the resulting vectors to recover an output matrix with the input’s shape. Practical GPU implementations can dispatch and combine tokens using vectorized scatter/gather operations. section 4.2.5
Combining expert outputs
For four tokens with eight-dimensional embeddings, the input and final output both have shape (4, 8). If a token selects experts E1 and E3 with weights 0.9 and 0.1, its output is 0.9 × E1(token) + 0.1 × E3(token). section 4.2.1 section 4.2.5
DeepSeekMoE adds fine-grained expert segmentation and shared expert isolation. Smaller routed experts can specialize more narrowly, while shared experts process every token and can learn knowledge needed across domains. The layer combines shared outputs with the weighted outputs of selected routed experts; a residual connection may be added at the layer or enclosing block. section 4.7 section 4.4.3 section 6.2.1
Figure 4.27 The complete forward pass of the DeepSeekMoE module, showing the three parallel data paths: the dense shared expert path, the sparse routed expert path with dynamic load balancing, and the residual connection.

Follow the shared, routed, and residual paths through the illustrated DeepSeekMoE module. section 4.5
Load imbalance can leave popular experts overloaded and other experts underused. Instead of adding an auxiliary balancing loss, the DeepSeek-style implementation adjusts a per-expert router bias from observed assignment counts. Underloaded experts receive a positive adjustment and overloaded experts a negative one; the implementation bounds the update with tanh. section 4.7 section 4.5
Listing 4.5 The update_bias Method for Load Balancing
@torch.no_grad()
def update_bias(self, x: torch.Tensor):
# Call once per optimizer step on the same tokens seen by forward.
# Uses the SAME logits (including current bias) to estimate loads.
N = x.shape[0] * x.shape[1]
logits = F.linear(x.reshape(-1,
self.d_model), self.centroids) + self.bias #A
_, idx = torch.topk(logits, self.top_k, dim=-1)
counts = torch.bincount(idx.flatten(),
minlength=self.n_routed).float() #B
avg = counts.sum() / max(1, self.n_routed)
# Smooth, bounded update; avoids large jumps from sign()
# violation > 0 => under-loaded (we want to increase its prior)
violation = (avg - counts) / (avg + 1e-6) #C
self.bias.add_(self.bias_lr * torch.tanh(violation)) #DInspect the count-based bias update used once per optimizer step in the standalone module. section 4.5
In one TinyStories experiment, models with nearly equal parameter counts were trained for 5,000 iterations. The standard MoE reached a best validation loss of 1.9854 in 14.29 minutes; DeepSeek-MoE reached 1.9451 in 11.67 minutes. These results describe that particular small-model comparison, not a universal performance guarantee. section 4.6









