liveBook · Manning

Mixture of experts

Also called MoE

A mixture-of-experts model is a neural-network architecture with specialized subnetworks and a routing mechanism that selects or combines experts for each input.

How 10 Manning books explain it

Rearchitecting LLMs

A mixture-of-experts block is a neural-network component containing multiple expert subnetworks and a router that selects a subset of them for each token.

How it is used
Uses dynamic routing to add clinical specialization to a dense model while checking general capabilities.
How deep it goes
Implements expert replacement, training, top-k routing, transplantation, and evaluation; contrasts the small-scale design with larger MoE models.
Examples
Builds clinical and general experts for SmolLM2, then extends the design with a Python-code expert and routing exercises.
What you can do after
Builds and evaluates an MoE by replacing MLPs, training experts and routers, and inspecting per-token routing.

Read how Rearchitecting LLMs explains mixture of experts →

Build a DeepSeek Model (From Scratch)

DeepSeek-MoE is a mixture-of-experts feed-forward architecture that routes inputs among specialized expert networks.

How it is used
Replaces dense Transformer feed-forward layers with sparsely routed experts to increase model capacity without proportional per-token computation.
How deep it goes
Works through routing mathematics, expert dispatch, bias updates, and a complete PyTorch module, while noting vectorized production dispatch.
Examples
Traces four eight-dimensional token embeddings through three experts and compares standard and DeepSeek-style MoE on TinyStories.
What you can do after
Build a DeepSeek-style MoE module from scratch, including shared experts, sparse routing, and dynamic load balancing.

Read how Build a DeepSeek Model (From Scratch) explains mixture of experts →

LLMs in Production

A model composed of multiple specialized expert subnetworks whose participation is selected or weighted for each input.

How it is used
Uses sparse expert activation to reduce computation in large language models while retaining specialization.
How deep it goes
Explains expert selection, specialization, and fine-tuning trade-offs, alongside a pretrained-model setup example.
Examples
Illustrates a decoder MoE, a reported 2-bit quantization result, and a pretrained Switch Transformer fine-tuning setup.
What you can do after
Evaluate MoE designs by comparing sparse computation, total parameter count, specialization, and fine-tuning requirements.

Read how LLMs in Production explains mixture of experts →

CUDA for LLMs

A mixture of experts is a neural-network architecture that routes inputs to selected specialized subnetworks called experts.

How it is used
Uses sparse expert routing in transformer inference to reduce per-token feed-forward computation.
How deep it goes
Traces gating, top-k selection, weighted expert outputs, CUDA operations, and validation; compares sequential and parallel selection tradeoffs.
Examples
Uses a PyTorch MoE layer, a CUDA-backed forward pass, and top-2 selection from eight scores.
What you can do after
Apply it via PyTorch and custom CUDA operations, then validate routing kernels and compare inference performance.

Read how CUDA for LLMs explains mixture of experts →

Introduction to Generative AI, Second Edition

A sparse mixture of experts is a mixture-of-experts architecture in which a router activates only a subset of expert modules for each input.

How it is used
Uses selective expert activation to make large language models more specialized and computationally efficient.
How deep it goes
Explains experts, routing, sparsity, and dense comparisons intuitively, but stops short of implementation details.
Examples
Uses an employee-specialization analogy, punctuation and parts-of-speech experts, and a three-expert layer visualization.
What you can do after
Evaluate and understand how sparse routing can increase model specialization without activating every parameter.

Read how Introduction to Generative AI, Second Edition explains mixture of experts →

Transformers in Action

A mixture-of-experts model routes inputs among specialized subnetworks called experts, typically activating or combining only a selected subset for each input.

How it is used
Uses expert routing to scale transformer language-model capacity without activating every expert for each token.
How deep it goes
Explains expert placement and routing, with top-k gating code and a GShard-style gating variant.
Examples
Illustrates expert selection with a PyTorch top-k gate and a GShard-style top-2 gate.
What you can do after
Evaluate and understand how sparse expert activation changes the relationship between model capacity and inference computation.

Read how Transformers in Action explains mixture of experts →

Machines that Think

A mixture-of-experts layer contains multiple specialized subnetworks and a gating mechanism that routes each input to a selected subset.

How it is used
Uses MoE layers to expand language-model capacity without proportionally increasing computation for each token.
How deep it goes
Explains routing and learned specialization, including training balance and execution costs, without implementation detail.
Examples
Illustrates routing with a family-doctor analogy and identifies Mixtral as a model using MoE.
What you can do after
Evaluate and understand how selective expert activation trades increased capacity against training and execution complexity.

Read how Machines that Think explains mixture of experts →

Vibe Engineering

A mixture-of-experts model is a neural-network architecture that selectively activates a subset of specialized parameter groups for each input.

How it is used
Guides local coding-model selection by balancing parameter quality, latency, and hardware fit.
How deep it goes
Explains sparse activation, resident memory, active computation, and KV-cache limits, but does not detail expert-routing internals.
Examples
Uses 80B/3B-active models, Qwen3-Coder-Next, and hardware thresholds from 16GB to 48GB-plus.
What you can do after
Operate local inference by matching MoE model size and context demands to available memory.

Read how Vibe Engineering explains mixture of experts →

Self-Hosting LLMs at Scale

A mixture of experts model is a model whose expert components are selectively activated for each token by a routing component.

How it is used
Uses the dense-versus-MoE distinction to inform self-hosted model and infrastructure choices.
How deep it goes
Explains token routing, expert selection, and the difference between stored parameters and computation per token.
Examples
Illustrates a 200-billion-parameter MoE activating 20 billion parameters per token and identifies gpt-oss MoE sizes.
What you can do after
Evaluate memory and computation trade-offs when sizing infrastructure for an MoE model.

Read how Self-Hosting LLMs at Scale explains mixture of experts →

Build Applications with Local AI Models on a Mac

A neural network architecture that routes inputs through selected expert subnetworks rather than activating every parameter for every input.

How it is used
Uses mixture of experts to compare model capacity and computing needs when selecting models for local use.
How deep it goes
Explains expert selection and joint training conceptually, without detailing routing algorithms or training procedures.
Examples
Compares GLM-5’s 744 billion total and 40 billion active parameters with Qwen3-Coder-Next’s 80 billion total and 3 billion active parameters.
What you can do after
Evaluate and understand model compute and memory requirements by distinguishing active parameters from total size and checking the selected build.

Read how Build Applications with Local AI Models on a Mac explains mixture of experts →

Also covered in

What to know first

Related concepts

Questions about mixture of experts

What is mixture of experts?

A mixture-of-experts model is a neural-network architecture with specialized subnetworks and a routing mechanism that selects or combines experts for each input.

Which Manning books explain mixture of experts?

Rearchitecting LLMs, Build a DeepSeek Model (From Scratch), LLMs in Production, CUDA for LLMs, Introduction to Generative AI, Second Edition, Transformers in Action, Machines that Think, Vibe Engineering and 2 more.

What should I know before learning mixture of experts?

Manning books that teach mixture of experts usually introduce feedforward network, token, transformer architecture and language model first.

What kinds of mixture of experts are there?

Kinds of mixture of experts include GLM-5 and hierarchical mixture of experts.

What are the parts of mixture of experts?

Mixture of experts includes expert network, router and routing.

What is mixture of experts also called?

Mixture of experts is also called MoE.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.