liveBook · Manning

Auxiliary-loss-free bias update

As explained in Build a DeepSeek Model (From Scratch)

An auxiliary-loss-free bias update adjusts expert routing biases from observed load deviations without adding an auxiliary balancing loss.

An auxiliary-loss-free bias update balances mixture-of-experts routing by adjusting expert biases from observed token loads, without adding a balancing loss to gradient training. Called once per optimizer step, it forms biased router logits, selects top-k experts, counts assignments, and applies a bounded correction that raises under-loaded experts and lowers over-loaded ones. section 4.5

The update follows a compact feedback procedure:

  • Add bias_lr multiplied by tanh(load violation), producing a smooth bounded correction for the next forward pass. section 4.5

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Implements logits, top-k selection, counting, violation calculation, and bounded updates, while stopping short of gradient-based balancing.
Examples
Uses a PyTorch update_bias method with torch.bincount, load violations, tanh smoothing, and configurable bias learning rate.
What it assumes
Assumes router logits, top-k expert selection, and routed dispatch; teaches load correction from observed assignments.
What you can do after
Operate/configure a PyTorch MoE router with dynamic expert-bias correction after each optimizer step.
How it connects
Connects expert routing, top-k selection, shared experts, sparse mixture-of-experts layers, and residual connections.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.