Dynamic bias adjustment balances a mixture-of-experts (MoE) router by changing per-expert biases in response to routing load. It avoids adding a balancing term to the training loss, where its strength would otherwise have to be traded against the main learning objective. section 4.4.4 section 4.4.4
The feedback loop operates at each training step:
- Add each expert’s bias to its raw router score before top-k expert selection. section 4.4.4
- Count the tokens routed to each expert and compare each load with the average. section 4.4.4
- Increase the bias for underloaded experts and decrease it for overloaded experts by a fixed step. The updated biases influence selection on the next step; they are not learned through backpropagation. section 4.4.4
The update uses the sign of the load violation to choose a fixed step’s direction. Defining the violation as average load minus expert load makes underloaded experts’ updates positive. section 4.4.4
Two experts
Suppose two experts receive 12 and 8 token assignments, for an average of 10. Decrease the first expert’s bias and increase the second’s by the same fixed step. On the next routing step, their adjusted scores make the overloaded expert less likely to be selected and the underloaded expert more likely. section 4.4.4
In one matched-parameter experiment, a standard MoE used an auxiliary balancing loss, while a DeepSeek-MoE used dynamic bias. For a validation batch, the standard model’s 22 routed experts showed hotspots, whereas the DeepSeek model’s 16 routed experts showed a nearly uniform distribution. The latter also had always-on shared experts, so the comparison is between model architectures, not an isolated bias-only change. section 4.6
Table 4.1 Comparison of training run
|
Model
|
Parameters (M)
|
Training Time (min)
|
Throughput (iter/min)
|
Best Val Loss
|
|
STANDARD_MOE
|
101.30
|
14.29
|
350.0
|
1.9854
|
|
DEEPSEEK_MOE
|
101.28
|
11.67
|
428.6
|
1.9451
|
The same experiment reports validation loss, training time, and throughput for both models; these are model-level results rather than a measurement of the bias mechanism alone. section 4.6
