liveBook · Manning

Expert network

As explained in Machines that Think

A specialized subnetwork within a mixture-of-experts layer that processes inputs when selected by the gating mechanism.

An expert network is a subnetwork in a mixture-of-experts (MoE) layer. The layer replaces a standard feedforward block with a pool of experts; a gating network routes each token representation to a small subset, whose weighted outputs are combined. Unselected experts stay inactive for that token. section 4.4.1

Experts are not assigned explicit competencies. Their specializations emerge through training: selected experts receive learning signals, while the gate learns which routing choices improve performance. Repeated routing and feedback reinforce useful divisions of responsibility. section 4.4.2

Selective activation brings both capacity gains and operational trade-offs:

  • Adding experts can increase model capacity without a proportional increase in computation per token. section 4.4.3
  • Uneven routing can leave some experts undertrained; varied token paths can also complicate batching, scheduling, and hardware utilization. section 4.4.2 section 4.4.3

Specialist referral

A family doctor directs cases to specialists; repeated referrals help the doctor route more effectively, while specialists improve in the cases they see most often. section 4.4.2

Figure 4.17 Structure of a Mixture of Experts layer. A gating network selects which experts should process each input and assigns a weight to their contributions. The chosen experts run in parallel, and their outputs are combined into a weighted sum. Experts that are not selected for a given input, such as the grayed-out Expert 3, remain inactive. By activating only a subset of experts at a time, the model increases its capacity without increasing the computation required at runtime.

The figure shows the gate selecting experts, the selected subnetworks processing an input in parallel, and their weighted outputs being combined. section 4.4.1

How Machines that Think covers it

How deep it goes
Defines expert selection and weighted combination, then traces specialization through training feedback.
Examples
Illustrates routing with a family-doctor analogy and cites Mixtral and, reportedly, GPT-4.
What you can do after
Understand how selective activation expands capacity while bounding per-token computation, and why routing creates workload trade-offs.
How it connects
Connects expert networks with mixture-of-experts layers, gating networks, feedforward layers, and Transformer architectures.

Also explained in

Compare how 5 books explain expert network →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.