Load imbalance occurs when tokens or computational work are distributed unevenly across expert modules or devices. In a mixture-of-experts layer, routing can create expert hotspots: some experts process disproportionately many tokens while others remain underutilized. Hotspots become computational bottlenecks, while unused experts leave capacity—and their potential knowledge—wasted. section 4.6 section 6.4.4
The imbalance may appear at different timescales. An auxiliary loss encourages balance across a long training run, but may not prevent severe skew within an individual batch. DeepSeek-MoE instead applies a dynamic bias that penalizes overloaded experts and rewards underloaded ones in real time, producing a more uniform batch-level workload. section 4.6
Routing and device examples
A validation batch routed through a standard MoE sends many more tokens to experts #7 and #17 than to experts #10 and #16. The corresponding DeepSeek-MoE distribution is remarkably uniform. In data parallelism, the main GPU gathers and aggregates gradients while other GPUs process their data slices, making the main device a bottleneck and producing poor scalability. section 4.6 section 6.4.4
Figure 4.29 Expert selection frequency for the Standard-MoE model in a sample batch. The uneven distribution highlights the problem of imbalanced routing.

Inspect the uneven expert-selection frequencies that expose routing imbalance in a sample standard-MoE batch. section 4.6
Figure 4.30 Expert selection frequency for the DeepSeek-MoE model in a sample batch. The distribution is remarkably uniform, demonstrating the effectiveness of the dynamic bias mechanism.

Compare the uniform expert-selection frequencies produced by dynamic bias in a sample DeepSeek-MoE batch. section 4.6
