Top-k gating routes each token to the experts with the highest router scores. A router maps each token’s hidden state to one logit per expert; selecting the top k logits and applying softmax to them gives normalized weights for the selected experts. section 9.3.1 section 9.4.2
The forward pass can use different routing behavior during training and inference:
- During training, compute softmax across all experts, run each expert on the full sequence, and combine outputs using their router weights. section 9.3.1
- During inference, select the top k experts per token, normalize their selected scores, and combine the selected outputs. Reorganizing assignments by expert lets each expert process its assigned tokens. section 9.3.1
Two of eight experts
Mixtral 8x7B uses eight experts and selects two per token. It has 46.7B total parameters and 12.9B active parameters; the routing mechanism uses softmax-normalized top-k logits. section 9.4.2
Figure 9.4 Two views of the same top-k routing decision. Left: the router produces one logit score per expert for each token; the two highest scores per row (highlighted) are selected. Right: the same assignments reorganized by expert, showing which tokens each expert will process and their renormalized weights — for each token, those weights sum to 1 across its assigned experts. The example uses four experts rather than two so that top_k = 2 leaves half the experts idle per token. The loop in TrainableTopKMoE.forward performs exactly this regrouping.
The figure shows how top-2 assignments across four experts are regrouped by expert, with selected weights summing to one for each token. section 9.3.1

