liveBook · Manning

Chapter 9: Dynamic routing with Mixture of Experts

What you will learn in chapter 9 of Rearchitecting LLMs

The introduction presents MoE as a way to add specialized experts to a dense model and defines the two routing strategies the chapter will compare.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The introduction presents MoE as a way to add specialized experts to a dense model and defines the two routing strategies the chapter will compare.

Key terms: Mixture of Experts, experts, dense model, domain expertise, soft routing, hard routing, router, token

9.1 Dynamic inference techniques

This section contrasts a conventional LLM’s uniform forward pass with three runtime approaches—early exiting, token dropping, and Mixture of Experts—and explains their inference-tool compatibility limits.

Key terms: forward pass, dynamic inference, runtime, early exiting, intermediate exit points, logits, confidence threshold, token dropping, attention weights, KV cache

9.2 Implementing and evaluating a two-expert MoE

Introduces the components and design of a two-expert MoE that replaces selected MLP modules while preserving the base model’s general behavior.

Key terms: experts, router, combined forward pass, routing weights, MoE Block, MLP module, layers 13 through 23, attention, Expert 0, Expert 1

9.2.1 Building the MoE

This section builds a two-expert MoE by replacing selected MLPs, preserving a general expert, and training a clinical expert and router on mixed-domain examples.

Key terms: Trainable2ExpertMoE, LlamaMLP, Expert 0, Expert 1, router, hidden state, logits, softmax, routing weights, FP32

9.2.2 Testing the MoE

This section tests whether a two-expert MoE preserves general responses, improves clinical extraction, and routes tokens toward the appropriate expert.

Key terms: Expert 0, Expert 1, schema compliance, valid JSON, forward pass, MoE Block, last_routing, routing weights, soft routing, selective activation

9.2.3 Implementing hard routing

This section changes a trained two-expert MoE to route each token through only one expert at inference, then examines how those decisions behave.

Key terms: hard routing, soft routing, inference, expert, evaluation mode, torch.argmax, router logits, expert_choice, boolean masks, hidden_states

9.2.4 How routing shapes model behavior

This section compares how soft and hard routing affect a MoE’s responses and cautiously interprets its general-benchmark results against a base model.

Key terms: forward functions, MoE Soft, MoE Hard, structured JSON, five general benchmarks, baseline, PIQA, clinical specialization

9.3 Reusing a fine-tuned model as an expert

The section explains why a clinical MLP from a previously fine-tuned model can be reused as an MoE expert, but may need adaptation to the base model’s attention layers.

Key terms: MLP Module, Expert 1, router, attention layers, clinical entity extraction

9.3.1 Creating a new MoE class

The section builds a configurable top-k MoE from the earlier two-expert class and explains how its forward pass uses all experts during training but only selected experts during inference.

Key terms: TrainableTopKMoE, num_experts, top_k, extra_experts, original MLP, deepcopy, routing layer, hidden_size, router logits, training mode

9.3.2 Transplant the expert and router-only training

This section builds a two-expert MoE from base and clinical MLPs, trains only its routers, and explains why the result falls short of the fine-tuned clinical model.

Key terms: MLP, Expert 0, Expert 1, TrainableTopKMoE, router, router-only training, frozen experts, learning rate, clinical entity extraction, compliance

9.3.3 Fine-tuning the expert tail

Explains why a transplanted clinical expert needs a small second training phase and reports its effects on schema compliance and general benchmarks.

Key terms: transplanted MLP Module, Attention Module, representational shift, expert tail, router, last six MLP layers, clinical expert, learning rates, schema compliance, top_k=1

9.4 From paper to practice

This section links the two earlier MoE implementations to research literature and introduces the papers discussed next.

9.4.1 Sparse upcycling

The section explains sparse upcycling and contrasts published results with a lower-budget implementation that freezes one expert and encourages router exploration.

Key terms: sparse upcycling, dense module, experts, Expert 0, general expert, router logits, Gaussian noise, specialization, training cost reduction, dense versions

9.4.2 Mixtral of Experts

This section compares Mixtral’s sparse MoE architecture with the chapter’s smaller implementation and contrasts their routing behavior.

Key terms: Mixtral 8x7B, MoE block, experts, top-k routing, active parameters, router, top-k logits, softmax, renormalized weights, clinical expert

9.5 Hands-on lab

9.6 Summary

Concepts explained in this chapter

← Chapter 8: Attention optimization Chapter 10: Exploring the transformer black box →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.