liveBook · Manning

Expert routing

As explained in Rearchitecting LLMs

Sparse routing processes each token with only a subset of available expert modules rather than with every expert.

Sparse expert routing reduces inference compute by selecting the top-k experts for each token. The router renormalizes weights across the selected experts, then groups tokens by expert so each module processes only its assigned tokens. section 9.3.1

A sparse forward pass follows these steps:

  • Score each token against the available experts and select its top-k scores. section 9.3.1
  • Recalculate softmax weights over selected experts; skip experts with no assigned tokens. section 9.3.1
  • Run each expert on its assigned tokens and add its weighted results to the output. section 9.3.1
Listing 9.10 TrainableTopKMoE signature
class TrainableTopKMoE(nn.Module):
    def __init__(self, original_mlp, hidden_size, num_experts=2, top_k=2,
                 extra_experts=None):
        ...
 
    def forward(self, hidden_states):
        ...

The routing diagram shows top-k selection and the regrouping of tokens by expert. section 9.3.1

How Rearchitecting LLMs covers it

How deep it goes
Implements top-k selection, weight renormalization, token regrouping, and expert execution, contrasting sparse inference with dense training.
Examples
Uses four experts with top_k=2, leaving half idle per token, and boolean masks to group assignments.
What you can do after
Build it from scratch by selecting experts per token, grouping tokens by expert, and combining weighted outputs.
How it connects
Connects mixture-of-experts layers with softmax gating, top-k selection, and token-to-expert dispatch.

Also explained in

Compare how 3 books explain expert routing →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.