chapter nine

9 Dynamic routing with Mixture of Experts

 

This chapter covers

  • Understanding dynamic routing for adaptive inference
  • The core components of a Mixture of Experts architecture (MoE)
  • Transforming an open source model into a MoE
  • Comparing routing strategies for inference
  • Reusing a specialized model as a MoE expert

Mixture of Experts has quickly moved from being a lab experiment to one of the most important architectures in frontier LLMs and, inevitably, it has made its way to SLMs. Mistral proved it could work outside the lab with the successful Mixtral 8x7B, and DeepSeek took the idea a step further, with fine-grained architectures: many small experts instead of a few large ones, each covering a very specific portion of the model's knowledge.

In this chapter, we apply the same logic but at a smaller scale. We will add highly specialized experts to a dense model, incorporating domain expertise without degrading, or even improving, its general capabilities, and you will get firsthand experience with an architecture already proven at the frontier, and one you'll be seeing much more of in SLMs.

9.1 Dynamic inference techniques

9.2 Implementing and evaluating a two-expert MoE

9.3 Building the MoE

9.4 Testing the MoE

9.5 Implementing hard routing

9.6 How routing shapes model behavior

9.7 Reusing a fine-tuned model as an expert

9.8 Creating a new MoE class

9.9 Transplant the expert and router-only training

9.10 Fine-tuning the expert tail

9.11 From paper to practice

9.12 Sparse upcycling

9.13 Mixtral of Experts

9.14 Hands-on lab

9.15 Summary