liveBook · Manning

Auxiliary-loss-free balancing

As explained in Build a DeepSeek Model (From Scratch)

Auxiliary-loss-free balancing is an expert-routing method that promotes balanced expert utilization without adding an auxiliary balancing loss.

Auxiliary-loss-free balancing keeps Mixture-of-Experts routing balanced without adding a balancing term to the main training objective. Traditional auxiliary losses create a trade-off: weak balancing permits overloaded experts, while strong balancing can interfere with language-learning performance. section 4.4.4 section 4.7

The method intervenes directly in routing. It adds a non-backpropagated bias to each expert’s raw router score before top-k selection, then updates that bias with a fixed step size γ after every training step. section 4.4.4

Implement the self-correcting update loop as follows:

  • Measure how many tokens each expert received in the current batch. section 4.4.4
  • Compare each expert’s load with the average to identify its load violation. section 4.4.4
  • Increase the bias of underloaded experts so the next router pass favors them. section 4.4.4
  • Decrease the bias of overloaded experts so the next router pass favors them less. section 4.4.4

Fixed-step bias update

For expert i, update the routing bias with bᵢ_new = bᵢ_old + u × sign(load_violation_error). The bias changes after the training step rather than through backpropagation. section 4.4.4

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains load measurement, violation detection, fixed-step bias updates, and score adjustment before top-k selection.
Examples
Uses token counts, average load, overloaded and underloaded experts, load violations, and a signed bias-update equation.
What it assumes
Assumes familiarity with MoE routing, experts, router scores, top-k selection, and training losses.
What you can do after
Build it from scratch by updating expert biases from batch loads and applying them before router top-k selection.
How it connects
Connects Mixture-of-Experts, expert routing, router scores, top-k selection, auxiliary loss, and load balancing.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.