liveBook · Manning

Gating weight

As explained in Build a DeepSeek Model (From Scratch)

A gating weight is a normalized routing coefficient that determines an input token's contribution from a selected expert.

A gating weight is a normalized routing coefficient that determines how strongly a selected expert contributes to an input token. In a mixture-of-experts router, softmax converts the original, bias-free router logits into the weights applied to the chosen experts. Load-balancing bias changes which experts win top-k selection, but does not distort the gating weights those experts receive. section 4.4.4

Implementation implications:

  • Compute gating weights from the original router scores. section 4.4.4
  • Add the adaptive expert bias only when selecting the top-k experts. section 4.4.4
  • Keep routing control separate from the contribution weights applied by selected experts. section 4.4.4

Balancing expert selection

If Experts 1 and 2 are underloaded, their biases increase; if Expert 3 is overloaded, its bias decreases. Future top-k selection shifts accordingly, while the selected experts still receive weights computed from the original logits. section 4.4.4

Figure 4.26 The bias term is added to the raw router logits before the top-k selection process.

Trace where the bias enters the router: before top-k selection, not before gating-weight computation. section 4.4.4

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains normalized weighting, softmax computation, top-k interaction, and the separation between routing bias and received weights.
Examples
Uses underloaded Experts 1 and 2 and overloaded Expert 3 to illustrate bias-directed routing changes.
What you can do after
Build it from scratch by separating bias-adjusted expert selection from softmax weights applied to selected experts.
How it connects
Connects gating weights to mixture-of-experts routing, router logits, softmax, top-k selection, and load balancing.

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.