Mixture of experts

As explained in Introduction to Generative AI, Second Edition

A sparse mixture of experts is a mixture-of-experts architecture in which a router activates only a subset of expert modules for each input.

A mixture of experts (MoE) divides a model into separate expert networks that specialize in handling different tokens or contexts. A gating network, also called a router, decides how much each expert contributes to the output. section 2.6

A sparse MoE activates only a few experts for each input, leaving many parameters inactive. This contrasts with a traditional transformer, which activates all parameters, and with a dense MoE, which combines every expert’s prediction. Selective activation aims to support larger or more specialized models without proportional computational costs, while addressing tradeoffs involving latency, speed, and expense. section 2.0 section 2.6

Routing work to specialists

The architecture resembles an organization that routes each task to employees with the strongest relevant skills. Model experts might specialize in punctuation or parts of speech, rather than in broad human subjects such as science or mathematics. section 2.6

Figure 2.6 A visualization of the Mixture of Experts layer of a neural network

A three-expert layer visualization makes sparsity concrete: only one shown expert contributes to the output. section 2.6

How Introduction to Generative AI, Second Edition covers it

How deep it goes

Explains experts, routing, sparsity, and dense comparisons intuitively, but stops short of implementation details.

Examples

Uses an employee-specialization analogy, punctuation and parts-of-speech experts, and a three-expert layer visualization.

What it assumes

Assumes only a basic understanding of model parameters and inputs while defining experts, routers, and sparsity.

What you can do after

Evaluate and understand how sparse routing can increase model specialization without activating every parameter.

How it connects

Connects mixture of experts with transformer models, gating networks, ensemble methods, sparse models, and large language models.

liveBookmanning.com

Read Introduction to Generative AI, Second Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  LLMs in ProductionA model composed of multiple specialized expert subnetworks whose participation is selected or weighted for each input.
• 

  liveBookmanning.com
  Transformers in ActionA mixture-of-experts model routes inputs among specialized subnetworks called experts, typically activating or combining only a selected subset for each input.
• 

  liveBookmanning.com
  Build Applications with Local AI Models on a MacA neural network architecture that routes inputs through selected expert subnetworks rather than activating every parameter for every input.

Compare how 4 books explain mixture of experts →

Related concepts

• expert networkhas part · An expert network is a specialized neural subnetwork within a mixture-of-experts model that processes inputs selected or routed to it by a gating mechanism.
• routinghas part · Routing is the selection of a path, destination, or processing component to direct an item, request, or flow toward an intended endpoint.
• feedforward networkalternative to · A feedforward network is a neural network or subnetwork in which information flows from inputs toward outputs through successive computations without recurrent connections or cycles.
• dense modelcontrasts with · A dense model is a neural model that applies the same computational pathway, rather than selectively routing each input to a subset of components.

This explainer was generated by Manning's knowledge graph from the text of Introduction to Generative AI, Second Edition. The section links point to the parts of the book it draws on.