Expert routing
Expert routing is the mechanism that selects which expert subnetworks process an input or token in a mixture-of-experts model.
CUDA for LLMs
Expert routing selects experts to process inputs in a mixture-of-experts model.
- How it is used
- Enables sparse expert activation during transformer inference by selecting a few experts per token.
- How deep it goes
- Traces routing from score normalization through weighted output combination and details a single-thread CUDA top-k scan.
- Examples
- Works through eight expert logits with top-2 selection and a CUDA-backed MoE forward pass.
- What you can do after
- Builds a simple CUDA top-k selection kernel for expert routing and applies it in an MoE forward pass.
Rearchitecting LLMs
Sparse routing processes each token with only a subset of available expert modules rather than with every expert.
- How it is used
- Reduces inference compute by sending each token only to its top-k experts.
- How deep it goes
- Implements top-k selection, weight renormalization, token regrouping, and expert execution, contrasting sparse inference with dense training.
- Examples
- Uses four experts with top_k=2, leaving half idle per token, and boolean masks to group assignments.
- What you can do after
- Build it from scratch by selecting experts per token, grouping tokens by expert, and combining weighted outputs.
Build a DeepSeek Model (From Scratch)
Routing is the mechanism that selects which expert subnetworks process a given input in a mixture-of-experts architecture.
- How it is used
- Uses learned token-to-expert selection to make mixture-of-experts computation sparse and specialized.
- How deep it goes
- Traces routing from token selection through expert outputs and matrix combination, without detailing router equations or training procedures.
- Examples
- Uses “What is 1+1?” and “The next day is.” alongside token-routing and expert-output matrix diagrams.
- What you can do after
- Evaluate how selective expert activation produces one Transformer-compatible output while avoiding unnecessary expert computation.
Read how Build a DeepSeek Model (From Scratch) explains expert routing →
What to know first
- feedforward networkA feedforward network is a neural network whose computations proceed from inputs toward outputs through directed connections without recurrent cycles or feedback.
- linear layerA linear layer is a neural-network layer that applies a learned affine transformation to its input, typically using weights and a bias.
- softmaxSoftmax is a function that exponentiates each value in a score vector and normalizes the results to produce positive values summing to one, forming a probability distribution.
- tokenA token is a discrete unit of text, such as a word, subword, punctuation mark, or symbol, produced by tokenization and processed by a language model.
Related concepts
- top-kuses · Top-k is a selection operation that retains the k items ranked highest according to a specified value or criterion.
- top-k kerneluses · The top-k kernel is the example CUDA implementation that extracts expert scores and indices for each token.
- mixture of expertsused by · A mixture-of-experts model is a neural-network architecture with specialized subnetworks and a routing mechanism that selects or combines experts for each input.
- top-k gatingproduced by · Top-k gating selects the k highest-scoring experts for each input and assigns routing weights to the selected experts.
- expert-choice routingincludes · Expert-choice routing is a routing strategy in which each expert selects the tokens it will process rather than receiving assignments from a token-side gate.
- expert networkuses · An expert network is a specialized neural subnetwork within a mixture-of-experts model that processes inputs selected or routed to it.
Questions about expert routing
What is expert routing?
Expert routing is the mechanism that selects which expert subnetworks process an input or token in a mixture-of-experts model.
Which Manning books explain expert routing?
CUDA for LLMs, Rearchitecting LLMs and Build a DeepSeek Model (From Scratch).
What should I know before learning expert routing?
Manning books that teach expert routing usually introduce feedforward network, linear layer, softmax and token first.
What kinds of expert routing are there?
Kinds of expert routing include expert-choice routing.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


