liveBook · Manning

Mixture of experts

As explained in Self-Hosting LLMs at Scale

A mixture of experts model is a model whose expert components are selectively activated for each token by a routing component.

When sizing infrastructure for a language model, parameter count alone does not describe the work done for each token. A mixture of experts (MoE) model divides its feed-forward layers into smaller components called experts. A router selects only a few experts for each token, and their outputs are combined. Attention remains dense. section 1.1.8

Figure 1.3 In a dense model, every token passes through the entire feed-forward layer of each transformer block. In an MoE model, attention remains dense, but in MoE transformer blocks the feed-forward layer is replaced by a set of experts. A router selects a small subset of them for each token (here, E1 and E5), and their outputs are combined. This allows the model to scale to many more parameters without activating all of them for every token. The experts typically need to reside in accelerator memory, possibly distributed across multiple GPUs.

The diagram contrasts dense feed-forward processing with routing to selected experts. section 1.1.8

The distinction matters for deployment: all experts must remain available in memory, even though only selected experts perform most of the computation for a token. Experts typically reside in accelerator memory and may be distributed across GPUs. Some tools can place them in CPU memory or on disk, but at a large speed cost. section 1.1.8 section 1.8

Sizing by stored and active parameters

A 200-billion-parameter MoE might activate only 20 billion parameters per token. It still needs all 200 billion available, but its computation per token may be closer to that of a much smaller dense model. Plan for the full memory requirement rather than sizing from the active parameter count alone. section 1.1.8

How Self-Hosting LLMs at Scale covers it

How deep it goes
Explains token routing, expert selection, and the difference between stored parameters and computation per token.
Examples
Illustrates a 200-billion-parameter MoE activating 20 billion parameters per token and identifies gpt-oss MoE sizes.
What you can do after
Evaluate memory and computation trade-offs when sizing infrastructure for an MoE model.
How it connects
Connects MoE to dense models, attention layers, feed-forward layers, inference cost, and GPU memory.

Also explained in

Compare how 10 books explain mixture of experts →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Self-Hosting LLMs at Scale. The section links point to the parts of the book it draws on.