liveBook · Manning

Dynamic routing bias

As explained in Build a DeepSeek Model (From Scratch)

A dynamic routing bias is a non-trainable per-expert adjustment to router scores that is updated to improve load balance.

Dynamic routing bias is a non-trainable per-expert adjustment added to router logits before top-k selection. Increasing an under-loaded expert’s bias makes it more likely to receive tokens, while decreasing overloaded experts’ relative advantage improves load balance. section 4.5

Figure 4.27 The complete forward pass of the DeepSeekMoE module, showing the three parallel data paths: the dense shared expert path, the sparse routed expert path with dynamic load balancing, and the residual connection.

The forward-pass diagram situates dynamic bias between router scoring and sparse expert dispatch, alongside shared experts and the residual connection. section 4.5

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains score adjustment, assignment counting, relative violation, bounded updates, and optimizer-step timing.
Examples
Uses PyTorch listings for router scoring, bias addition, expert assignment counting, and bounded bias updates.
What it assumes
Assumes familiarity with mixture-of-experts layers, router logits, top-k selection, and expert dispatch.
What you can do after
Operate/configure a dynamic bias update with per-expert counts, bias_lr, and optimizer-step synchronization.
How it connects
Connects mixture-of-experts routing with expert centroids, top-k routing, shared experts, residual connections, and load balancing.

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.