liveBook · Manning

Top-k gating

As explained in Rearchitecting LLMs

Top-k routing selects the k experts with the highest router scores for each token and normalizes their selected weights.

Top-k gating routes each token to the experts with the highest router scores. A router maps each token’s hidden state to one logit per expert; selecting the top k logits and applying softmax to them gives normalized weights for the selected experts. section 9.3.1 section 9.4.2

The forward pass can use different routing behavior during training and inference:

  • During training, compute softmax across all experts, run each expert on the full sequence, and combine outputs using their router weights. section 9.3.1
  • During inference, select the top k experts per token, normalize their selected scores, and combine the selected outputs. Reorganizing assignments by expert lets each expert process its assigned tokens. section 9.3.1

Two of eight experts

Mixtral 8x7B uses eight experts and selects two per token. It has 46.7B total parameters and 12.9B active parameters; the routing mechanism uses softmax-normalized top-k logits. section 9.4.2

Figure 9.4 Two views of the same top-k routing decision. Left: the router produces one logit score per expert for each token; the two highest scores per row (highlighted) are selected. Right: the same assignments reorganized by expert, showing which tokens each expert will process and their renormalized weights — for each token, those weights sum to 1 across its assigned experts. The example uses four experts rather than two so that top_k = 2 leaves half the experts idle per token. The loop in TrainableTopKMoE.forward performs exactly this regrouping.

The figure shows how top-2 assignments across four experts are regrouped by expert, with selected weights summing to one for each token. section 9.3.1

How Rearchitecting LLMs covers it

How deep it goes
Implements training and inference routing, including expert selection and output weighting.
Examples
Uses a four-expert top-2 routing diagram and Mixtral 8x7B’s eight-expert, top-2 configuration.
What you can do after
Build it from scratch by selecting top-k router logits, normalizing their weights, and combining selected expert outputs in a PyTorch forward pass.
How it connects
Connects top-k gating to mixture-of-experts models, router logits, and softmax.

Also explained in

Compare how 2 books explain top-k gating →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.