liveBook · Manning

Dynamic bias adjustment

As explained in Build a DeepSeek Model (From Scratch)

Dynamic bias adjustment is a feedback-based technique that changes per-expert routing biases according to measured routing loads.

Dynamic bias adjustment balances a mixture-of-experts (MoE) router by changing per-expert biases in response to routing load. It avoids adding a balancing term to the training loss, where its strength would otherwise have to be traded against the main learning objective. section 4.4.4 section 4.4.4

The feedback loop operates at each training step:

  • Add each expert’s bias to its raw router score before top-k expert selection. section 4.4.4
  • Count the tokens routed to each expert and compare each load with the average. section 4.4.4
  • Increase the bias for underloaded experts and decrease it for overloaded experts by a fixed step. The updated biases influence selection on the next step; they are not learned through backpropagation. section 4.4.4
bᵢ_new = bᵢ_old + u * sign(load_violation_error)

The update uses the sign of the load violation to choose a fixed step’s direction. Defining the violation as average load minus expert load makes underloaded experts’ updates positive. section 4.4.4

Two experts

Suppose two experts receive 12 and 8 token assignments, for an average of 10. Decrease the first expert’s bias and increase the second’s by the same fixed step. On the next routing step, their adjusted scores make the overloaded expert less likely to be selected and the underloaded expert more likely. section 4.4.4

In one matched-parameter experiment, a standard MoE used an auxiliary balancing loss, while a DeepSeek-MoE used dynamic bias. For a validation batch, the standard model’s 22 routed experts showed hotspots, whereas the DeepSeek model’s 16 routed experts showed a nearly uniform distribution. The latter also had always-on shared experts, so the comparison is between model architectures, not an isolated bias-only change. section 4.6

Table 4.1 Comparison of training run
Model
Parameters (M)
Training Time (min)
Throughput (iter/min)
Best Val Loss
STANDARD_MOE
101.30
14.29
350.0
1.9854
DEEPSEEK_MOE
101.28
11.67
428.6
1.9451

The same experiment reports validation loss, training time, and throughput for both models; these are model-level results rather than a measurement of the bias mechanism alone. section 4.6

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains load measurement, fixed-step bias updates, and their placement before top-k selection, then compares expert utilization empirically.
Examples
Compares a 22-routed-expert standard MoE with a 16-routed-expert DeepSeek-MoE on a validation batch and matched-parameter training runs.
What you can do after
Evaluate and understand how load-driven bias updates influence expert selection and how their model-level results compare with auxiliary-loss balancing.
How it connects
Connects mixture-of-experts routing with top-k selection, auxiliary loss, and load balancing.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.