liveBook · Manning

Load imbalance

As explained in Build a DeepSeek Model (From Scratch)

Load imbalance is an uneven distribution of tokens or computational work across expert modules.

Load imbalance occurs when tokens or computational work are distributed unevenly across expert modules or devices. In a mixture-of-experts layer, routing can create expert hotspots: some experts process disproportionately many tokens while others remain underutilized. Hotspots become computational bottlenecks, while unused experts leave capacity—and their potential knowledge—wasted. section 4.6 section 6.4.4

The imbalance may appear at different timescales. An auxiliary loss encourages balance across a long training run, but may not prevent severe skew within an individual batch. DeepSeek-MoE instead applies a dynamic bias that penalizes overloaded experts and rewards underloaded ones in real time, producing a more uniform batch-level workload. section 4.6

Routing and device examples

A validation batch routed through a standard MoE sends many more tokens to experts #7 and #17 than to experts #10 and #16. The corresponding DeepSeek-MoE distribution is remarkably uniform. In data parallelism, the main GPU gathers and aggregates gradients while other GPUs process their data slices, making the main device a bottleneck and producing poor scalability. section 4.6 section 6.4.4

Figure 4.29 Expert selection frequency for the Standard-MoE model in a sample batch. The uneven distribution highlights the problem of imbalanced routing.

Inspect the uneven expert-selection frequencies that expose routing imbalance in a sample standard-MoE batch. section 4.6

Figure 4.30 Expert selection frequency for the DeepSeek-MoE model in a sample batch. The distribution is remarkably uniform, demonstrating the effectiveness of the dynamic bias mechanism.

Compare the uniform expert-selection frequencies produced by dynamic bias in a sample DeepSeek-MoE batch. section 4.6

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains batch-level routing skew, long-run auxiliary loss, dynamic bias, and centralized gradient aggregation, without deriving implementation code.
Examples
Uses expert-frequency charts, hotspot experts, DeepSeek-MoE routing, and a centralized nn.DataParallel gradient workflow.
What you can do after
Diagnose routing or device bottlenecks and evaluate auxiliary-loss and dynamic-bias approaches to workload balancing.
How it connects
Connects mixture-of-experts, expert routing, auxiliary loss, dynamic bias, data parallelism, gradient aggregation, and scalability.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.