Chapter 9: Dynamic routing with Mixture of Experts
The introduction presents MoE as a way to add specialized experts to a dense model and defines the two routing strategies the chapter will compare.
After this chapter you will be able to
- Specialized experts in MoE. Explain that an MoE can use experts focused on particular portions of a model’s knowledge, and that the chapter aims to add domain expertise to a dense model while preserving its general capabilities.
- Soft and hard routing. Distinguish soft routing from hard routing in the proposed two-expert system: soft routing gives each token a weighted mix of both experts, while hard routing makes a binary choice.
- Uniform versus dynamic computation. Explain why a conventional LLM does the same work to generate each token regardless of how difficult the input is, and how dynamic inference changes that pattern.
- Token wise expert routing. Distinguish Mixture of Experts from exiting early or dropping tokens, and describe the router's role.
- Expert routing and output mixing. Explain how the experts, router, and combined forward pass produce one MoE output. Describe how the router assigns weights based on the token being processed.
- Selected MLP module replacement. Describe where the two-expert MoE blocks are inserted and what remains unchanged. Explain the section’s reason for selecting the deepest layers.
- Copied experts and frozen weights. Explain how the two experts are initialized and which weights are trained. Justify how copying pretrained weights and freezing the original expert serve the design.
- Two-expert block construction. Describe how Trainable2ExpertMoE wraps an existing LlamaMLP, preserves it as Expert 0, copies it to create Expert 1, and adds a token-level router.
- Soft routing and expert combination. Trace a token’s hidden state through router logits, softmax routing weights, and the weighted combination of the two expert outputs.
- Early routing collapse prevention. Explain why Gaussian noise is added to router logits during training and how it gives the trainable expert time to specialize.
- Selective replacement and freezing. Describe which MLPs are replaced in this experiment and which resulting parameters remain trainable, including why those choices were made.
- Mixed-domain routing examples. Justify using interleaved clinical and simple general-conversation examples to train the router.
- General and clinical test results. Describe how the MoE performs on a general knowledge prompt and a clinical extraction prompt, and state what the dataset-level schema compliance results show.
- Token-level routing evidence. Explain how the section inspects routing weights and describe the different expert preferences observed for general and clinical tokens.
- Soft routing and activation cost. Distinguish the two-expert soft routing used here from selective expert activation, including their implications for inference cost.
- Hard versus soft routing. Distinguish hard routing from soft routing in this MoE: at inference, the router selects one expert per token rather than combining both experts’ outputs, without retraining the experts or router.
- Single-expert forward pass. Trace how the evaluation-mode forward pass chooses an expert, separates tokens with masks, and restores their outputs to the original sequence positions.
- Contextual routing interpretation. Explain why the observed expert choice is not an inherent property of an isolated token, and distinguish the observed clinical routing tendency from proof of internal specialization.
- Routing and response length. Explain how the same MoE weights can produce different responses under soft and hard forward functions. Describe what the Barcelona control prompt shows and the tentative explanation for the shorter output.
- General benchmark trade-offs. Describe the pattern across the five general-knowledge benchmarks and explain how cautiously the section interprets it.
- Clinical expert reuse constraints. Explain why the Chapter 7 model’s MLP is a candidate MoE expert and why training only the router may not suffice when that MLP is paired with the base model’s attention layers.
- Configurable expert construction. Describe how TrainableTopKMoE assembles its experts and what num_experts, top_k, and extra_experts specify.
- Dense training output. Trace how the training-mode forward pass combines expert outputs and state why this implementation omits Gaussian noise.
- Sparse inference execution. Trace how inference selects experts for each token, redistributes tokens for processing, and combines their results.
- Clinical expert transplantation. Describe how the existing clinical MLPs become experts in the MoE alongside the base model’s MLPs.
- Router-only training choice. Explain why this phase freezes both experts and trains only the router, and how that differs from the earlier two-expert training.
- Clinical compliance and mismatch. State the router-only model’s clinical extraction result and explain the proposed reason it falls short of the Chapter 7 model.
- Transplanted expert mismatch. Explain why the transplanted clinical MLP needs adaptation when used in the MoE, and why the layers near the output are targeted instead of retraining the expert from scratch.
- Expert tail training phase. Describe the second training phase: which weights are trainable, how long they are trained, and why the learning rates are low.
- Sparse upcycling principle. Explain how sparse upcycling initializes MoE experts and why the same idea can be used with different model architectures.
- Low-budget expert adaptation. Distinguish the paper's expert training from the book's lower-budget approach, including the purposes of freezing Expert 0 and adding noise to router logits.
- Mixtral sparse expert computation. Explain how Mixtral replaces each MLP with an eight-expert MoE block but selects only two experts per token, reducing active computation relative to its total parameter count.
- Syntactic versus domain routing. Distinguish the syntactic specialization observed in Mixtral from the domain distinction the chapter’s clinical-expert router needed to learn.
What it builds on
- General behaviour of a Transformer model (chapter 3) · Assumes familiarity with a Transformer model's forward processing.
- What the KV cache is and why it matters (chapter 8) · Assumes the reader has encountered the KV cache and its stored key-value pairs.
- The classic MLP mechanism (chapter 3) · Assumes familiarity with the MLP module that experts replace.
- The Transformer dimensions: depth and width (chapter 3) · Transformer blocks provide the depth dimension used to choose replacement layers.
- QLoRA training on the clinical dataset (chapter 7) · The earlier clinical training dataset and prompt format are reused here.
- Evaluating the results against our goals (chapter 7) · The earlier evaluation and schema compliance metric used for clinical extraction.
Section by section
Introduction
The introduction presents MoE as a way to add specialized experts to a dense model and defines the two routing strategies the chapter will compare.
- Specialized experts in MoE: Explain that an MoE can use experts focused on particular portions of a model’s knowledge, and that the chapter aims to add domain expertise to a dense model while preserving its general capabilities.
Why it matters: This is the motivation for adding experts rather than replacing the dense model outright. - Soft and hard routing: Distinguish soft routing from hard routing in the proposed two-expert system: soft routing gives each token a weighted mix of both experts, while hard routing makes a binary choice.
Why it matters: This distinction gives the reader a basis for following the chapter’s routing comparison.
Key terms: Mixture of Experts, experts, dense model, domain expertise, soft routing, hard routing, router, token
9.1 Dynamic inference techniques
This section contrasts a conventional LLM’s uniform forward pass with three runtime approaches—early exiting, token dropping, and Mixture of Experts—and explains their inference-tool compatibility limits.
- Uniform versus dynamic computation: Explain why a conventional LLM does the same work to generate each token regardless of how difficult the input is, and how dynamic inference changes that pattern.
Why it matters: This contrast explains why the section considers conditional computation at all. - Confidence based early exiting: Describe how an intermediate exit point uses confidence to decide whether processing should stop before the remaining layers.
Why it matters: It shows one way dynamic inference avoids processing layers that may not be needed. - Attention guided token dropping: Trace how token dropping selects tokens to omit from later layers and what happens to the KV cache.
Why it matters: It illustrates dynamic inference that reduces work by changing which tokens continue through the model. - Token wise expert routing: Distinguish Mixture of Experts from exiting early or dropping tokens, and describe the router's role.
Why it matters: Routing to experts is the dynamic-inference approach developed in the rest of this chapter. - Inference library compatibility limits: Explain why runtime decisions complicate use with standard inference libraries and qualify MoE's relative maturity.
Why it matters: An inference technique's potential benefit does not guarantee that standard serving tools can run it without adaptation.
Key terms: forward pass, dynamic inference, runtime, early exiting, intermediate exit points, logits, confidence threshold, token dropping, attention weights, KV cache
9.2 Implementing and evaluating a two-expert MoE
Introduces the components and design of a two-expert MoE that replaces selected MLP modules while preserving the base model’s general behavior.
- Expert routing and output mixing: Explain how the experts, router, and combined forward pass produce one MoE output. Describe how the router assigns weights based on the token being processed.
Why it matters: These three roles explain what the MoE block does before its implementation is examined. - Selected MLP module replacement: Describe where the two-expert MoE blocks are inserted and what remains unchanged. Explain the section’s reason for selecting the deepest layers.
Why it matters: The placement defines the scope of the architectural change and why those layers were chosen. - Copied experts and frozen weights: Explain how the two experts are initialized and which weights are trained. Justify how copying pretrained weights and freezing the original expert serve the design.
Why it matters: The initialization and freezing choices let the model pursue a clinical specialty without training the whole model or discarding its general behavior.
Key terms: experts, router, combined forward pass, routing weights, MoE Block, MLP module, layers 13 through 23, attention, Expert 0, Expert 1
9.2.1 Building the MoE
This section builds a two-expert MoE by replacing selected MLPs, preserving a general expert, and training a clinical expert and router on mixed-domain examples.
- Two-expert block construction: Describe how Trainable2ExpertMoE wraps an existing LlamaMLP, preserves it as Expert 0, copies it to create Expert 1, and adds a token-level router.
Why it matters: The copied starting point lets one expert specialize while the original remains available for general behavior. - Soft routing and expert combination: Trace a token’s hidden state through router logits, softmax routing weights, and the weighted combination of the two expert outputs.
Why it matters: This explains how routing changes a block’s output without changing the output magnitude expected by the rest of the model. - Early routing collapse prevention: Explain why Gaussian noise is added to router logits during training and how it gives the trainable expert time to specialize.
Why it matters: Without early exploration, the clinical expert could receive too little signal to become useful. - Selective replacement and freezing: Describe which MLPs are replaced in this experiment and which resulting parameters remain trainable, including why those choices were made.
Why it matters: Selective replacement and freezing make specialization feasible without retraining the entire model or discarding its general expert. - Mixed-domain routing examples: Justify using interleaved clinical and simple general-conversation examples to train the router.
Why it matters: The router needs a clear, sustained contrast between domains to learn when the clinical expert is appropriate. - Mixed-precision training choices: Explain how the training setup balances memory use, stable updates, and different learning needs for the router and clinical expert.
Why it matters: These choices let the selected expert and routers train on limited hardware without losing small weight updates.
Key terms: Trainable2ExpertMoE, LlamaMLP, Expert 0, Expert 1, router, hidden state, logits, softmax, routing weights, FP32
9.2.2 Testing the MoE
This section tests whether a two-expert MoE preserves general responses, improves clinical extraction, and routes tokens toward the appropriate expert.
- General and clinical test results: Describe how the MoE performs on a general knowledge prompt and a clinical extraction prompt, and state what the dataset-level schema compliance results show.
Why it matters: Testing both domains checks that clinical specialization did not come at the expense of ordinary responses. - Token-level routing evidence: Explain how the section inspects routing weights and describe the different expert preferences observed for general and clinical tokens.
Why it matters: The routing weights show whether expert preference changes with the content of individual tokens, rather than relying only on final answers. - Soft routing and activation cost: Distinguish the two-expert soft routing used here from selective expert activation, including their implications for inference cost.
Why it matters: It separates the specialization demonstrated by this test from the efficiency benefit of activating only selected experts.
Key terms: Expert 0, Expert 1, schema compliance, valid JSON, forward pass, MoE Block, last_routing, routing weights, soft routing, selective activation
9.2.3 Implementing hard routing
This section changes a trained two-expert MoE to route each token through only one expert at inference, then examines how those decisions behave.
- Hard versus soft routing: Distinguish hard routing from soft routing in this MoE: at inference, the router selects one expert per token rather than combining both experts’ outputs, without retraining the experts or router.
Why it matters: You can explain the intended inference saving without mistaking hard routing for a new training procedure. - Single-expert forward pass: Trace how the evaluation-mode forward pass chooses an expert, separates tokens with masks, and restores their outputs to the original sequence positions.
Why it matters: The masks explain how sparse processing still produces a complete, correctly ordered output. - Preserved routing analysis and weights: Describe how the hard-routing block replaces the earlier MoE blocks while retaining trained weights, and how its recorded decisions remain usable by the existing routing analysis.
Why it matters: This separates a change in routing behavior from a change in what the trained blocks contain. - Contextual routing interpretation: Explain why the observed expert choice is not an inherent property of an isolated token, and distinguish the observed clinical routing tendency from proof of internal specialization.
Why it matters: It keeps routing observations from being mistaken for fixed keyword rules or direct evidence about an expert’s internals.
Key terms: hard routing, soft routing, inference, expert, evaluation mode, torch.argmax, router logits, expert_choice, boolean masks, hidden_states
9.2.4 How routing shapes model behavior
This section compares how soft and hard routing affect a MoE’s responses and cautiously interprets its general-benchmark results against a base model.
- Routing and response length: Explain how the same MoE weights can produce different responses under soft and hard forward functions. Describe what the Barcelona control prompt shows and the tentative explanation for the shorter output.
Why it matters: A changed answer length does not necessarily mean the model changed its account of the building. - General benchmark trade-offs: Describe the pattern across the five general-knowledge benchmarks and explain how cautiously the section interprets it.
Why it matters: The benchmark pattern provides broader evidence than one prompt without warranting a definitive claim.
Key terms: forward functions, MoE Soft, MoE Hard, structured JSON, five general benchmarks, baseline, PIQA, clinical specialization
9.3 Reusing a fine-tuned model as an expert
The section explains why a clinical MLP from a previously fine-tuned model can be reused as an MoE expert, but may need adaptation to the base model’s attention layers.
- Clinical expert reuse constraints: Explain why the Chapter 7 model’s MLP is a candidate MoE expert and why training only the router may not suffice when that MLP is paired with the base model’s attention layers.
Why it matters: It explains both the appeal of reusing a specialized expert and the compatibility problem that reuse creates.
Key terms: MLP Module, Expert 1, router, attention layers, clinical entity extraction
9.3.1 Creating a new MoE class
The section builds a configurable top-k MoE from the earlier two-expert class and explains how its forward pass uses all experts during training but only selected experts during inference.
- Configurable expert construction: Describe how TrainableTopKMoE assembles its experts and what num_experts, top_k, and extra_experts specify.
Why it matters: The expert list determines which existing and specialized MLPs the router can use. - Per-token router scores: Explain what the router produces from a token’s hidden state and how its output relates to the available experts.
Why it matters: The same per-token scores feed both the dense training path and the selective inference path. - Dense training output: Trace how the training-mode forward pass combines expert outputs and state why this implementation omits Gaussian noise.
Why it matters: Training uses a weighted contribution from every expert, rather than the sparse execution used at inference. - Sparse inference execution: Trace how inference selects experts for each token, redistributes tokens for processing, and combines their results.
Why it matters: Separating the routing decision from masked expert execution explains where sparse inference actually saves compute. - Optional routing records: Describe what last_routing records and why it need not be retained when expert traffic is not being inspected.
Why it matters: Routing records support inspection without being required for the model’s output.
Key terms: TrainableTopKMoE, num_experts, top_k, extra_experts, original MLP, deepcopy, routing layer, hidden_size, router logits, training mode
9.3.2 Transplant the expert and router-only training
This section builds a two-expert MoE from base and clinical MLPs, trains only its routers, and explains why the result falls short of the fine-tuned clinical model.
- Clinical expert transplantation: Describe how the existing clinical MLPs become experts in the MoE alongside the base model’s MLPs.
Why it matters: The two experts bring existing capabilities; the new router must learn how to use them. - Router-only training choice: Explain why this phase freezes both experts and trains only the router, and how that differs from the earlier two-expert training.
Why it matters: Freezing the reused experts isolates whether routing alone can make their combined model effective. - Clinical compliance and mismatch: State the router-only model’s clinical extraction result and explain the proposed reason it falls short of the Chapter 7 model.
Why it matters: The result shows that reusing a trained expert does not necessarily preserve its original performance when its surrounding modules change.
Key terms: MLP, Expert 0, Expert 1, TrainableTopKMoE, router, router-only training, frozen experts, learning rate, clinical entity extraction, compliance
9.3.3 Fine-tuning the expert tail
Explains why a transplanted clinical expert needs a small second training phase and reports its effects on schema compliance and general benchmarks.
- Transplanted expert mismatch: Explain why the transplanted clinical MLP needs adaptation when used in the MoE, and why the layers near the output are targeted instead of retraining the expert from scratch.
Why it matters: It explains why reusing the specialist calls for a targeted adjustment rather than starting over. - Expert tail training phase: Describe the second training phase: which weights are trainable, how long they are trained, and why the learning rates are low.
Why it matters: The phase preserves most of the existing specialist while adapting it to the new model. - Specialization evaluation results: State the reported schema-compliance result for both routing settings and describe the mixed but generally reassuring general-benchmark results.
Why it matters: Checking both specialist behavior and general benchmarks shows whether the adaptation achieved its purpose without an obvious broad trade-off.
Key terms: transplanted MLP Module, Attention Module, representational shift, expert tail, router, last six MLP layers, clinical expert, learning rates, schema compliance, top_k=1
9.4 From paper to practice
This section links the two earlier MoE implementations to research literature and introduces the papers discussed next.
9.4.1 Sparse upcycling
The section explains sparse upcycling and contrasts published results with a lower-budget implementation that freezes one expert and encourages router exploration.
- Sparse upcycling principle: Explain how sparse upcycling initializes MoE experts and why the same idea can be used with different model architectures.
Why it matters: This is the shared starting point behind the paper's models and the book's implementation. - Low-budget expert adaptation: Distinguish the paper's expert training from the book's lower-budget approach, including the purposes of freezing Expert 0 and adding noise to router logits.
Why it matters: The changes explain how the book sought useful specialization with less data and compute, and the trade-off that creates. - Published results and limits: State what the paper reported about upcycled models and explain why its training-cost figures cannot be claimed for the book's implementation.
Why it matters: It separates the paper's measured benefits from what the book's modified experiment actually establishes.
Key terms: sparse upcycling, dense module, experts, Expert 0, general expert, router logits, Gaussian noise, specialization, training cost reduction, dense versions
9.4.2 Mixtral of Experts
This section compares Mixtral’s sparse MoE architecture with the chapter’s smaller implementation and contrasts their routing behavior.
- Mixtral sparse expert computation: Explain how Mixtral replaces each MLP with an eight-expert MoE block but selects only two experts per token, reducing active computation relative to its total parameter count.
Why it matters: It explains how a large expert model can keep per-token inference computation comparatively low. - Shared routing logic across scales: Describe how Mixtral and the chapter’s two-expert MoE share the same essential selection logic despite differing in scale and in the use of a transplanted clinical expert.
Why it matters: The smaller implementation illustrates routing logic also used in a much larger MoE. - Syntactic versus domain routing: Distinguish the syntactic specialization observed in Mixtral from the domain distinction the chapter’s clinical-expert router needed to learn.
Why it matters: Expert routing does not necessarily organize itself by subject area; its behavior depends on the experts and setup.
Key terms: Mixtral 8x7B, MoE block, experts, top-k routing, active parameters, router, top-k logits, softmax, renormalized weights, clinical expert
9.5 Hands-on lab
9.6 Summary
Concepts explained in this chapter
- mixture of expertsA mixture-of-experts block is a neural-network component containing multiple expert subnetworks and a router that selects a subset of them for each token.
- routerA request router is a component that directs incoming requests to an appropriate processing queue or model configuration.
- expert networkAn expert is an independently parameterized neural-network component in a mixture-of-experts architecture that processes selected inputs.
- hard routingHard routing is a routing behavior that assigns a token to selected experts without continuously weighting all candidate expert outputs.
- routing weightA routing weight is a scalar produced by a mixture-of-experts router that controls an expert's contribution to the combined output.
- freezeSelective freezing is a training technique that keeps chosen model parameters fixed while allowing other parameters to update.
- Gaussian noiseGaussian noise is a random perturbation whose values follow a normal probability distribution.
- top-k gatingTop-k routing selects the k experts with the highest router scores for each token and normalizes their selected weights.
- soft routingSoft routing is a mixture-of-experts routing strategy that combines the outputs of multiple experts using continuous routing weights.
- sparse upcyclingSparse Upcycling is a method for converting a pretrained dense model into a sparse mixture-of-experts model, commonly by duplicating feed-forward components as experts and further training the resulting architecture.
- dynamic inferenceDynamic inference is model execution that makes input-dependent runtime decisions about which computations, tokens, or submodels to use.
- expert routingSparse routing processes each token with only a subset of available expert modules rather than with every expert.
- torch.topktorch.topk is a PyTorch operation that returns the largest or smallest k elements of a tensor together with their indices.
- multilayer perceptronA multilayer perceptron is a feed-forward neural network composed of successive learned linear transformations and nonlinear activations.
- small language modelA small language model is a language model with comparatively limited parameters and computational requirements, typically optimized for efficient deployment.
- hidden stateA hidden state is an internal vector representation produced by a neural model layer for the information processed up to that layer.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
