Mixture of experts
A mixture-of-experts model is a neural-network architecture with specialized expert subnetworks and a routing mechanism that selects or combines experts for each input.
LLMs in Production
A model composed of multiple specialized expert subnetworks whose participation is selected or weighted for each input.
- How it is used
- Uses sparse expert selection to reduce inference computation while retaining specialization across diverse inputs.
- How deep it goes
- Explains transformer-layer replacement, routing and finetuning implications, but does not specify a router implementation.
- Examples
- Shows a gate-and-router decoder diagram and finetuning of a pretrained Switch Transformer on a local text dataset.
- What you can do after
- Apply it via Hugging Face Transformers by finetuning a pretrained Switch Transformer, or evaluate sparse routing as an architectural trade-off.
Introduction to Generative AI, Second Edition
A sparse mixture of experts is a mixture-of-experts architecture in which a router activates only a subset of expert modules for each input.
- How it is used
- Uses selective expert activation to make large language models more specialized and computationally efficient.
- How deep it goes
- Explains experts, routing, sparsity, and dense comparisons intuitively, but stops short of implementation details.
- Examples
- Uses an employee-specialization analogy, punctuation and parts-of-speech experts, and a three-expert layer visualization.
- What you can do after
- Evaluate and understand how sparse routing can increase model specialization without activating every parameter.
Read how Introduction to Generative AI, Second Edition explains mixture of experts →
Transformers in Action
A mixture-of-experts model routes inputs among specialized subnetworks called experts, typically activating or combining only a selected subset for each input.
- How it is used
- Uses expert routing to scale transformer LLM capacity while limiting active computation per token.
- How deep it goes
- Explains FFN replacement and top-k routing, then shows a PyTorch gate that returns selected experts and scores.
- Examples
- Shows an MoE architecture diagram, a PyTorch top-k gate, and a GShard-style top-2 gating listing.
- What you can do after
- Build it from scratch: implement a PyTorch top-k gate that selects expert indices and normalizes their scores.
Read how Transformers in Action explains mixture of experts →
Build Applications with Local AI Models on a Mac
A neural network architecture that routes inputs through selected expert subnetworks rather than activating every parameter for every input.
- How it is used
- Clarifies why a large model can use fewer parameters per token than its total size suggests.
- How deep it goes
- Explains routing, expert activation, and joint training in plain language, without implementation details.
- Examples
- Uses GLM-5’s 744B-total/40B-active contrast and Qwen3-Coder-Next’s 80B-total/3B-active specifications.
- What you can do after
- Evaluate/understand MoE model specifications by separating per-token computation from download size and RAM needs.
Read how Build Applications with Local AI Models on a Mac explains mixture of experts →
Related concepts
- expert networkhas part · An expert network is a specialized neural subnetwork within a mixture-of-experts model that processes inputs selected or routed to it by a gating mechanism.
- routinghas part · Routing is the selection of a path, destination, or processing component to direct an item, request, or flow toward an intended endpoint.
- feedforward networkalternative to · A feedforward network is a neural network or subnetwork in which information flows from inputs toward outputs through successive computations without recurrent connections or cycles.
- dense modelcontrasts with · A dense model is a neural model that applies the same computational pathway, rather than selectively routing each input to a subset of components.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.