liveBook · Manning

Routing bias

As explained in Build a DeepSeek Model (From Scratch)

A routing bias is a persistent per-expert score adjustment that influences expert selection without being learned through backpropagation.

A routing bias is a persistent per-expert score adjustment added to raw router scores before top-k selection. It influences expert choice without being learned through backpropagation. section 4.4.4

Use the bias as a self-correcting load-balancing loop:

  • Measure each expert’s token load after a training step. section 4.4.4
  • Increase the bias of underloaded experts and decrease the bias of overloaded experts. section 4.4.4
  • Apply the fixed-step update for the next training step, raising or lowering future selection scores. section 4.4.4

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains load measurement, directional updates, fixed-step timing, and pre-selection score adjustment, but not implementation code.
Examples
Uses expert-load calculations, load-violation diagrams, a signed bias-update equation, and a pre-selection routing diagram.
What it assumes
Assumes router scores, top-k selection, expert loads, and training steps, while grounding the bias mechanism from those operations.
What you can do after
Apply it by adjusting per-expert scores from measured overload or underload without adding an auxiliary balancing loss.
How it connects
Connects mixture-of-experts routing with router logits, top-k selection, expert load balancing, and backpropagation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.