liveBook · Manning

Mixture of experts

As explained in Machines that Think

A mixture-of-experts layer contains multiple specialized subnetworks and a gating mechanism that routes each input to a selected subset.

A mixture of experts (MoE) expands a model’s capacity by letting different inputs use different subsets of its parameters. Unlike a dense layer, which applies the same full computation to every token, an MoE layer activates only selected subnetworks for each token. This can increase total capacity without increasing per-token computation proportionally. section 4.4.1

In a Transformer, an MoE layer can replace a feedforward block with a pool of independent subnetworks called experts. A gating network scores the token’s representation, selects a small subset of experts, and assigns weights to their contributions. The selected experts run in parallel, and their outputs are combined; unselected experts remain inactive for that token. section 4.4.1 section 4.4.2

Figure 4.17 Structure of a Mixture of Experts layer. A gating network selects which experts should process each input and assigns a weight to their contributions. The chosen experts run in parallel, and their outputs are combined into a weighted sum. Experts that are not selected for a given input, such as the grayed-out Expert 3, remain inactive. By activating only a subset of experts at a time, the model increases its capacity without increasing the computation required at runtime.

Follow a token through the gate, selected experts, and weighted output. section 4.4.1

Specialization is learned rather than assigned by hand. Selected experts receive a direct learning signal from their contribution to the loss, while the gate adjusts its routing as expert choices affect performance. Repeated assignments can make experts better at particular patterns and make those assignments more useful. section 4.4.2

Routing requires a balance. If too few experts receive most tokens, the others may not learn useful behavior; if tokens are spread too evenly, specialization can weaken. Selective computation also complicates execution: tokens in the same batch may take different paths, creating uneven device workloads and making batching and scheduling less efficient. section 4.4.2 section 4.4.3

MoE layers are typically used sparingly within Transformer stacks, while attention layers remain unchanged. This combines shared processing with selective additional capacity rather than replacing the underlying architecture. Mixtral is one example of a model using this approach. section 4.4.3 section 4.4.1

How Machines that Think covers it

How deep it goes
Explains routing and learned specialization, including training balance and execution costs, without implementation detail.
Examples
Illustrates routing with a family-doctor analogy and identifies Mixtral as a model using MoE.
What you can do after
Evaluate and understand how selective expert activation trades increased capacity against training and execution complexity.
How it connects
Connects MoE to Transformer feedforward blocks, attention, gating networks, and training loss.

Also explained in

Compare how 10 books explain mixture of experts →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.